Bulk-populate a DataLad dataset from CSV, TSV, or JSON URL manifests.
npx skills add https://github.com/bcmcpher/my-skills --skill datalad-addurls
Please help me install this Agent Skill. Skill: datalad-addurls Source: https://github.com/bcmcpher/my-skills/tree/main/plugins/datalad-cli/skills/datalad-addurls Command: npx skills add https://github.com/bcmcpher/my-skills --skill datalad-addurls
Bulk-populate a DataLad dataset by reading a URL manifest and adding each file as an annexed entry with its download URL registered as the annex remote. DataLad will fetch content on demand using the datalad get command.
Use this skill to bulk-import files into a DataLad dataset by providing a manifest of URLs.
High-intent search queries and answers about installing and using this skill.
To bulk import files into a DataLad dataset, read a URL manifest and add each file as an annexed entry with its download URL registered as the annex remote. DataLad will then fetch content on demand using the datalad get command.
You can use CSV, TSV, and JSON manifests to populate a DataLad dataset. The process parses the manifest, detects URL and filename columns, and constructs the datalad addurls command for large-scale file ingestion.
After adding URLs from a manifest, DataLad registers each download URL as an annex remote. Content is not downloaded immediately; instead, DataLad fetches files on demand when you run the datalad get command.
Yes, you can use parallel downloads when ingesting files into a DataLad dataset. The workflow supports options for parallel downloads and missing-value handling when building the datalad addurls command from your manifest.
To add URLs to a DataLad annex, your manifest must contain parseable URL and filename columns. The workflow also requires a meaningful commit message to properly register the annexed entries during the bulk ingestion process.
bcmcpherDrop or remove annexed files from local DataLad datasets.
bcmcpherFetch annexed content and initialize subdatasets in DataLad datasets.
bcmcpherConfigure and manage DataLad dataset siblings and their remotes.
frxiaobeiFetch article images and generate a manifest.json with original URLs and local paths.
