CPython

CPython General

A minimal CPython model contains three essential files, the model specification file model.spec.json, a python model file (default name: model.py, ), and __init__.py in order to define the model as python module.

directory structure

model/
├── __init__.py
├── model.py
└── model.spec.json

Model Specification File

The model specification file is used to define the entrypoint and some other options for the model calculation.

model.spec.json

{
    "entry_file": "model.py",
    "entry_function_name": "execute",

    "resource_files": [
        "**/*.csv",
        "**/*.xlsx",
        "**/*.txt",
        "**/*.json",
    ],

    "docker_container_recreate": false
}

• entry_file defines the name of the entry point filename of the python model. (default: model.py)

• entry_function_name defines the name of the function that is executed for each task item context. (default: execute)

• resource_files defines resource files that will be included in the model. (default: only .py files)

• docker_container_recreate always creates a fresh docker container for each new task (default: false)

Python Model

The minimal setup for a CPython model is Python file model.py that contains a entry point function execute(context) for the work item calculation.

Example model.py

import numpy as np

def execute(ctx):
    ctx.logger.info(f'ENTER {__file__}')
    
    # create ouput
    ctx.data.output['output_var_1'] = ctx.data.input['input_var_1']
    
    # complete workitem with success
    ctx.outcome.success()

    ctx.logger.info(f'EXIT {__file__}')

Context API

Note

This Calculation Context API will later be delivered as a separate python module for local model testing and syntax completion.

The context argument from the execute(context) entry point contains all the information in order to calculate a task item (work item).

Logging

Logging will be captured on the executing task trial result data as python.runner.log.

import logging

def execute(context):

    # use context logger
    # See: https://docs.python.org/3/library/logging.html#module-logging
    context.logger.debug('debug message ...')
    context.logger.info('info message ...')
    context.logger.warning('warning message ...')
    context.logger.error('error message ...')
    context.logger.critical('critical message ...')

    # you are also able to create your own logger which will log the the initial logging config
    my_logger = logging.getLogger('custom_logger')
    my_logger.info('hello from custom_logger')

    # direct logging
    logging.info('hello from root logger')

Task Item Id & Trial

def execute(context):

    # access to task item (work item) id and  execution trial
    current_id = context.id
    context.logger.info(f'Current Id: {current_id}')  # e.g. '1.1.1'

    current_trial = context.trial
    context.logger.info(f'Current Trial: {current_trial}')  # e.g. 1

Variable Specification

def execute(context):

    # access variable specification (input, output)
    for spec in context.spec.input:
        context.logger.info(f'Input Variable: {spec.name}, {spec.uom}')

    for spec in context.spec.output:
        context.logger.info(f'Output Variable: {spec.name}, {spec.uom}')

    # you also can access them by attributes or by indexed attributes
    spec = context.spec.input.variable_1
    context.logger.info(f'Input Variable: {spec.name}, {spec.uom}')

    spec = context.spec.input['variable_1']
    context.logger.info(f'Input Variable: {spec.name}, {spec.uom}')

    spec = context.spec.output.variable_1
    context.logger.info(f'Output Variable: {spec.name}, {spec.uom}')

    spec = context.spec.output['variable_1']
    context.logger.info(f'Output Variable: {spec.name}, {spec.uom}')

Data Input & Output

def execute(context):

    # read and write data
    input1 = context.data.input.variable_1
    input2 = context.data.input['variable_2']

    context.data.output.variable_1 = input1 + input2
    context.data.output['variable_2'] = input1 - input2

Execution Outcome

def execute(context):

    # setting the outcome for your calculation
    # the outcome it will be undefined by default
    context.outcome.undefined()
    context.outcome.error()
    context.outcome.warning()
    context.outcome.success()

Resource Files

It is possible to include resource files (e.g. csv, xslx, json, txt, etc.) within your model.

Caution

Keep your model as small as possible! Do not include resource files that are not required in your calculation (e.g. broad wildcard includes). Larger models can drastically reduce runtime performance because of operations like decompressing, writing model data, etc.

example_model/
├── data
│   ├── data1.csv
│   └── data2.xlsx
├── __init__.py
├── model.spec.json
├── my_model.py
└── requirements.txt

Example model.spec.json:

Include CSV and XLSX resource files

{
    "entry_file": "my_model.py",
    "entry_function_name": "my_entry_function",
    "resource_files": [
        "data/data1.csv",
        "data/data2.xlsx"
    ]
}

The resource_files property accepts file globbing (includes & excludes) from the root directory where the model.spec.json file is placed. Exclusions are prefixed with ! (e.g. !**/*.pyc), see file globbing: https://docs.python.org/3/library/glob.html.

CPython 3.8

Type Specifier: Python.CPython3.8

in addition to the common model specification for the CPython worker it is possible to include a standard requirements.txt. All dependencies/modules from this requirements.txt file will be available in your model.py.

Caution

Include only modules that are actually used! Each module dependency will be installed by pip. This can be a time consuming process depending on module size and complexity and will increases the overall runtime.

directory structure

model/
├── __init__.py
├── model.py
├── model.spec.json
└── requirements.txt

Requirements File

If your model has module dependencies, for example numpy, pandas or anything else from the public PyPi index (https://pypi.org/search/), you can include a requirements.txt file.

See: https://pip.pypa.io/en/stable/cli/pip_install/#requirement-specifiers.

requirements.txt

numpy==1.19.1
pandas
matplotlib

You have to include the requirements.txt file in your model.spec.json in the resource_files section in order to be processed by pip.

Additional model spec settings (model.spec.json)

{
    "resource_files": [
        "requirements.txt"
    ]
}

CPython Generic

Type Specifier: Python.Generic

The CPython generic worker can be used to run python models with custom python docker images (e.g. anaconda). These images are special docker CPython images and build and managed by ENEXSA.

To use a custom image you have to provide a docker_image in the model.spec.json file.

Additional model spec settings (model.spec.json)

{
    "docker_image": "myrepo/my_custom_image:latest"
}

Enexsa Anaconda Image 2021.11

The Anaconda docker image contains a full Anaconda 2021.11 installation for the calculation and requires a environment.yml file. This environment.yml is used to setup your specific conda environment.

See Conda Environments: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Anaconda 2021.11 Image: nexus.enexsa.com/enexsahub/calculation-cpython-conda:2021.11-enx1.2.0

directory structure

model/
├── __init__.py
├── model.py
├── model.spec.json
└── environment.yml

Additional model spec settings (model.spec.json)

{
    "resource_files": [
        "environment.yml"
    ],

    "docker_image": "nexus.enexsa.com/enexsahub/calculation-cpython-conda:2021.11-enx1.2.0",
}

Environment File (YAML)

Note

Channel Limits! Only the anaconda main channel and proxy channels from nexus.enexsa.com are currently allowed to use in the environment.yml, see example below.

environment.yml

name: enexsa
channels:
- https://nexus.enexsa.com/repository/conda-forge   # conda-forge proxy channel (optional)
- https://nexus.enexsa.com/repository/conda-cantera # cantera proxy channel (optional)
- defaults # default anaconda channel
dependencies:
- python=3.8.12
- numpy=1.20.1
- pandas