My Experience with Prefect so Far (and a quick getting started guide)

Overall initial impressions of Pros of Prefect

Getting Started with Prefect

sudo pip install prefect
import pandas or whatever
from prefect import task
from prefect import Flow
@task # this is the prefect identifier
def function1(X):
@task
def function2(Y):
with flow(name='Test Flow') as flow:
function1results = function1(conn)
function2(function1results)
python prefect_test.py
prefect run -p prefect_test.py
sudo apt update
sudo apt install python3-pip
prefect backend cloud
prefect auth login --key <YOUR-KEY>
pip install supervisor
[unix_http_server]
file=/tmp/supervisor.sock ; the path to the socket file
[supervisord]
loglevel=debug ; log level; default info; others: debug,warn,trace
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
[program:prefect-agent]
command=prefect agent local start
from prefect.tasks.secrets import PrefectSecret@task()
def function1(test_key):
someapi.api_key = test_key['key']

with flow(name="test_flow") as flow:
test_key = PrefectSecret("Test")
conn = snowflake.connector.connect(
user = snowflake_creds['user'],
password = snowflake_creds['password']
account = snowflake_creds['account'],
warehouse = snowflake_creds['warehouse'],
database = snowflake_creds['database'],
schema = snowflake_creds['schema'],
role = snowflake_creds['role'])
cur = conn.cursor()cur.execute()
prefect register -p test_script.py --project TestProject

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store