skills.rest
logicalclocks
logicalclocksOFFICIAL
Explore all skills by this author →
Website𝕏@hopsworksGitHub

hops-job

Automate Hopsworks job and Airflow workflow orchestration and execution.

15|32|Updated Nov 13, 2021
Search Tags:#python#data processing#airflow#job orchestration#hopsworks
One-click install
npx skills add https://github.com/logicalclocks/hopsworks-api --skill hops-job
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hops-job
Source: https://github.com/logicalclocks/hopsworks-api/tree/main/skills/hops/hops-job
Command: npx skills add https://github.com/logicalclocks/hopsworks-api --skill hops-job

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires hopsworks, airflow, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill streamlines the process of creating, configuring, scheduling, and running Hopsworks jobs or Airflow workflows, reducing manual steps and improving efficiency.

Core Features & Use Cases

  • Job Orchestration: Automate the scheduling and execution of Hopsworks jobs and Airflow workflows.
  • Script Deployment: Upload and execute Python/PySpark scripts from HopsFS.
  • Environment Configuration: Select and utilize pre-defined Python environments for job execution.
  • Use Case: If you have a series of data processing tasks in Hopsworks that need to be executed in a specific order, this Skill can automate the entire workflow.

Quick Start

Deploy a new Hopsworks job by running the following command:

hops job deploy feature-pipeline feature_pipeline.py --env python-feature-pipeline --cron @daily --run --wait --overwrite

Frequently Asked Questions about hops-job

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate Hopsworks job orchestration for Python scripts?▼

Hopsworks job orchestration is automated by scheduling and executing Python or PySpark scripts directly from HopsFS. This Skill handles job creation, environment configuration, and cron scheduling to streamline the entire workflow execution process.

Can I use Airflow workflows to execute PySpark scripts in Hopsworks?▼

Yes, Airflow workflows can execute PySpark scripts within Hopsworks. This Skill automates the orchestration and execution of these workflows, handling the deployment and configuration of scripts uploaded from HopsFS.

Do I need a pre-configured Hopsworks environment to run scheduled data processing jobs?▼

Yes, you need Hopsworks and Airflow properly set up and configured to run scheduled data processing jobs. This Skill requires these platforms to handle job creation, environment selection, and script execution from HopsFS.

What is the best way to deploy a Hopsworks feature pipeline with a daily schedule?▼

The best way to deploy a Hopsworks feature pipeline daily is using the job deploy command. You specify the script, select a pre-defined Python environment, set the cron schedule, and execute with run and wait flags.

How does Hopsworks job scheduling handle environment configuration for data processing tasks?▼

Hopsworks job scheduling handles environment configuration by allowing users to select and utilize pre-defined Python environments for job execution. This ensures data processing tasks run with the correct dependencies.

Why are my Airflow workflows not executing Hopsworks jobs properly?▼

Airflow workflows fail to execute Hopsworks jobs properly when dependencies are missing. Ensure Hopsworks and Airflow are correctly configured, and that Python or PySpark scripts are properly uploaded and accessible from HopsFS.

Related Skills

Explore More Software Engineering Skills→
logicalclockslogicalclocks
15

hops-app

Deploy Streamlit and custom Python apps on Hopsworks from HopsFS or Git.

Official
Intermediate
tanyagraytanyagray

gitlab-ci

Manage GitLab CI/CD pipelines and jobs using the glab CLI.

Community
Intermediate
logicalclockslogicalclocks
15

hops-batch-inference

Automate batch inference in Hopsworks using Python or PySpark.

Official
Advanced
melikhanmutlumelikhanmutlu

airflow-dag-patterns

Automate the design and deployment of production-grade Airflow DAGs.

Community
Advanced