Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci to update all literature monthly #134

Merged
merged 4 commits into from
Dec 31, 2024
Merged

ci to update all literature monthly #134

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b65663c
ci to update all literature monthly
hdashnow Dec 30, 2024
a0cc3f1
Update data
hdashnow Dec 30, 2024
aa14e1f
fix workflow triggers
hdashnow Dec 31, 2024
a50316a
fix PR action trigger
hdashnow Dec 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 27 additions & 4 deletions .github/workflows/update-loci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ name: Update data

permissions:
contents: write
pull-requests: write

on:
pull_request:
Expand All @@ -11,6 +12,8 @@ on:
- "scripts/**"
- "workflow/**"
- ".github/workflows/**"
schedule:
- cron: "45 2 1 * *" # Run monthly at 2:45 am
workflow_dispatch:

jobs:
Expand Down Expand Up @@ -41,18 +44,38 @@ jobs:
conda info

- name: Check loci and update bed files (short)
#if: ${{ github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch'}}
if: ${{ github.event_name == 'pull_request'}}
run: snakemake --config stages="skip-refs"

# Enable this step to open an SSH session for debugging
# - name: SSH debug
# uses: mxschmitt/action-tmate@v3

- name: Find new literature (intermediate)
if: ${{ github.event_name == 'workflow_dispatch' }}
run: |
conda init
source /home/runner/.bashrc
conda activate strchive
snakemake --config stages="new-refs"

- name: Update all literature (long)
if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the long and intermediate steps will both run on workflow dispatches? Just checking that this is what you intended.

Copy link
Contributor

@vincerubinetti vincerubinetti Dec 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I made this comment originally is because, the way they're named "short intermediate long", I would expect them to be mutually exclusive, i.e. only one of them runs each workflow run. But I haven't looked inside the scripts to see what's actually going on yet, is that actually the case? It doesn't have to be; I'd just recommend removing the lengths from the names if not.

Looks like now there's a clear split/association between the three trigger types (pr, schedule, manual) and the three script lengths (short, intermediate, long). Note that schedule and manual will also trigger PR on subsequent commits. Here's a breakdown of the scenarios (in terms of automatic workflow runs):

  • Someone commits to main → nothing happens
  • Someone manually creates a PR → someone commits → pull_request triggers, "short" script runs → updated data committed to PR
  • Someone manually runs workflow_dispatch → "intermediate" script runs → PR opened with first commit of updated data → someone commits again to opened PR → pull_request triggers → "short" script runs → updated data committed to PR
  • schedule runs once a month → "long" script runs → PR opened with first commit of updated data → someone commits again to opened PR → pull_request triggers → "short" script runs → updated data committed to PR

If that looks okay, then the PR LGTM

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for breaking that down!
Yes, I think that is the desired behavior. Let's see if that all works out in practice.

run: |
conda init
source /home/runner/.bashrc
conda activate strchive
snakemake

- name: Open pull request with updated files
if: ${{ !(github.event_name == 'pull_request') }}
if: ${{ !(github.event_name == 'pull_request') || github.event_name == 'schedule' }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This "or schedule" is also unnecessary as it will already be covered by "not pull request"

uses: peter-evans/create-pull-request@v7
with:
branch: update-data
title: Update data

- name: Commit updated files to current pull request
if: ${{ github.event_name == 'pull_request' }}
if: ${{ github.event_name == 'pull_request' || github.event_name == 'schedule' }}
vincerubinetti marked this conversation as resolved.
Show resolved Hide resolved
uses: stefanzweifel/git-auto-commit-action@v5
with:
commit_message: Update data
commit_message: Update data
2 changes: 2 additions & 0 deletions scripts/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@ dependencies:
- jsbeautifier
- statsmodels
- plotly
- zlib
- r-essentials
- r-base
- r-jsonlite
- r-dplyr
- r-rentrez
Expand Down
26 changes: 21 additions & 5 deletions scripts/get-literature.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,26 @@
# More useful error messages
options(error=traceback)

## Set default CRAN mirror
local({r <- getOption("repos")
r["CRAN"] <- "https://cran.r-project.org"
options(repos=r)
})

# Install biomaRt if not already installed
if (!requireNamespace("biomaRt", quietly = TRUE)) {
if (!requireNamespace("BiocManager", quietly = TRUE)) {
install.packages("BiocManager")
tryCatch({
if (!requireNamespace("biomaRt", quietly = TRUE)) {
if (!requireNamespace("BiocManager", quietly = TRUE)) {
install.packages("BiocManager", ask = FALSE, update = FALSE)
}
BiocManager::install("biomaRt", ask = FALSE)
}
BiocManager::install("biomaRt")
}
}, error = function(e) {
cat("Error installing biomaRt package.\n", file=stderr())
quit(status = 1)
})

tryCatch({
suppressPackageStartupMessages({
library(jsonlite)
library(dplyr)
Expand All @@ -21,6 +33,10 @@ suppressPackageStartupMessages({
library(stringr)
library(purrr)
})
}, error = function(e) {
cat("Error loading required packages.\n", file=stderr())
quit(status = 1)
})

### Data Setup
args <- commandArgs(trailingOnly = TRUE)
Expand Down
2 changes: 2 additions & 0 deletions scripts/setup-miniconda-patched-environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ dependencies:
- jsbeautifier
- statsmodels
- plotly
- zlib
- r-essentials
- r-base
- r-jsonlite
- r-dplyr
- r-rentrez
Expand Down