def module::add_data | ( | i | ) |
Add data Input: { cm_module_uoa - module UOA cm_data_uoa - data UOA (cm_data_uid) - force data UID (if UOA is alias) (cm_repo_uoa) - repo UOA (use_default_repo) - if 'yes', use default repo (cm_console) - by default, do not output anything except errors if 'txt' then output as plain text if 'web' then output for web (html, image, etc) if 'json' then output as json (cm_file_upload) - text content of file for upload (cm_file_upload_base64) - file content as base64.urlsafe_b64encode (cm_file_upload_name) - preffered uploaded file name (cm_file_upload_type) - type of file (if .zip, it will be automatically unziped) (cm_archive) - 'none' 'already_archived_with_internal_name' - should save file under default cM name (cm_archive.zip) 'already_archived_with_user_name' - should save file under user name (cm_file_upload_tmp_uid) - normally generated automatically by web server when uploading files cm_array - use ONLY this array to add data (cm_admin) - if 'yes', override access control - only internal use, can not be used during web access (explicitly removed) } Output: { cm_return - return code >0 if error -1 if entry exists cm_path - path to created data entry cm_uid - uid (from UOA) cm_alias - alias (from UOA) cm_uoa - alias or uid if alias=='' }
Add module Input: { } Output: { cm_return - return code >0 if error cm_path - path to created data entry }
Add repository Input: { (path) - path to create repository (otherwise current) or URL path_type - absolute, relative_to_cm, remote_url (cm_repo_uoa) - repo UOA where to create entry (cm_array) - additional description (repo_alias) - alias on the disk (repo_name) - user friendly name (when listing repositories) (repo_uid) - set UID for the repository - useful when creating personal user repository in shared environments ... } Output: { cm_return - return code >0 if error cm_path - path to created data entry ... - other paramaters from core add_data function }
Add user (and public person profile) Input: { cm_username - username (UOA) cm_user_password - user password cm_user_password_repeat - repeat user password for checking name - user real name (will be used as alias) (cm_array) - Note, that if cm_array is present, all above parameters should be inside it (cm_repo_uoa) - repo UOA where to create user entry } Output: { cm_return - return code >0 if error cm_path - path to created data entry ... - other paramaters from core add_data function }
def module::add_data_through_web | ( | i | ) |
Add/update data through web form Input: { browse_module_uoa - module UOA browse_data_uoa - data UOA (browse_repo_uoa) - repo UOA (cm_back_json) - for back button (cm_only_note_like) - if 'yes', update only note and ranking cm_mode - 'add' or 'update' to reuse the same function cm_array - data array ... } Output: { cm_return - return code >0 if error -1 if entry exists If entry was created (not first web form), then: cm_path - path to created data entry cm_uid - uid (from UOA) cm_alias - alias (from UOA) }
def module::all | ( | i | ) |
ReIndex all repos Input: { (index_repo_uoa) - repo uoa to prune reindexing (index_module_uoa) - module uoa to prune reindexing (index_data_uoa) - data uoa to prune reindexing (can include pattern * and ?) (delay) - delay after indexing individual entry (sequential) - if 'yes' perform sequential index (very slow, but reports errors) } Output: { cm_return - return code >0 if error }
Test all Input: { } Output: { cm_return - return code >0 if error }
def module::analyze | ( | i | ) |
def module::analyze_variation | ( | i | ) |
Analyze variation in characteristics Input: { (data) - list of data entries to analyze (prune_by_summary_point_relaxed) - if data is not provided, search will be performed for all entries; returned entries can be pruned by this dictionary using 'summary_point_relaxed' in entries characteristic_key - key to get characteristic (will be converted to float) percent_threshold - percent threshold to detect benchmarks where characteristic has high variability (cm_repo_uoa) - working repository } Output: { cm_return - if =0, success }
def module::any | ( | i | ) |
def module::browse | ( | i | ) |
cM basic repository browsing Input: { (browse_module_uoa) - module UOA (browse_data_uoa) - data UOA (browse_repo_uoa) - repo UOA (cm_subaction_delete) - delete entry (cm_subaction_repo_delete) - delete repository (cm_subaction_view) - view entry (show_add_info) - additional view config (show additional info about entry) (show_param_names) - additional view config (show parameter names (for development)) (cm_subaction_module_view) - view module (cm_subaction_repo_view) - view repository (cm_subaction_update) - update entry (cm_subaction_repo_update) - update repository (cm_subaction_add) - add entry (cm_subaction_module_add) - add module (cm_subaction_repo_add) - add repository (cm_button_reset) - reset current view (cm_back_json) - previous web state to create 'back' button } Output: { cm_return - return code >0 if error }
def module::build | ( | i | ) |
Build program Input: { See code.run (all parameters will be passed to code.run) (work_dir_repo_uoa) - change to the working directory in the repository (repo UOA) (work_dir_module_uoa) - change to the working directory in the repository (module UOA) (work_dir_data_uoa) - change to the working directory in the repository (data UOA) build_target_os_uoa - target OS uoa for building If the above parameters are set, the following parameters will be added to run_set_env2 from the given cM entry: CM_SOURCE_FILES="source_files" CM_SOURCE_FILES_WITHOUT_EXT="source_files" (extension will be removed) CM_TARGET_FILE="target_file" (target_file) - output file after building (add_target_extension) - take extension from run_target_os_uoa (include_deps) - add 'include' directory from this list (include_prefix) - add prefix before include directory (usually -I or /I) (compiler_vars) - add compiler vars (compiler_var_prefix) - add compiler var prefix (usually -D or /D) (lib_deps) - add libraries from this list (use CM_TARGET_FILE) (compilation_type) - 'static' (by default) or 'dynamic' (install) - if 'yes', install to cM entry (install_data_uid) - if !='', it will be used for installation data entry (install_data_alias) - if !='', use it for installation data entry (install_data_display_as_alias) - use this display as alias for a generated entry (install_module_uoa) - if =='', module 'code' is used (install_repo_uoa) - repo for installation entry (source_files) - list of source files; if empty, use list of source files from the program entry (add_rem_to_script) - add rem to install script (add_to_code_entry) - add array to code entry (ignore_script_error) - if 'yes', do not generate error when script returns !=0 (needed for experiment pipelines to continue running) (cm_dependencies) - dependencies that set all other dependencies (code/include/lib). Format [{"class UOA":"code UOA"},{},...] (skip_target_file_check) - if 'yes', skip target file check (run_cmd_out1) - pass to 'code run' to redirect output1 (run_cmd_out2) - pass to 'code run' to redirect output2 (clean_program) - if 'yes', clean program before (clean_program_script_name) - script name for cleaning ... (code_deps) - list with code UOA for dependencies [{"index":"uoa"} ...] (run_set_env2) - array with environment variables to be set before compiling (cm_verbose) - if 'yes', print all info } Output: { cm_return = 0, if successful Parameters from "code run" when executing script target_file - output file after building target_file_size - size of output file }
Build model Input: { model_module_uoa - model's module (model_data_uoa) - model's data (if need to record) model_name - (earth, svm) data ct_dimensions_input ct_dimensions_output desc (record_data_to_file_prefix) - if !='', use this filename prefix instead of randomly generated } Output: { cm_return - return code >0 if error }
Build model Input: { data ct_dimensions_input ct_dimensions_output desc model_name - (earth, svm) (record_data_to_file_prefix) - if !='', use this filename prefix instead of randomly generated } Output: { cm_return - return code >0 if error file_with_model - file with model }
def module::calculate_similarity | ( | i | ) |
Calculate similarity FGG: for now simple euclidean distance - it's just a demo of the concept, since it has many limitations so we need to collaboratively extend it (using PCA, SVM, etc) Input: { program1_milepost_features - MILEPOST features for program one program2_milepost_features - MILEPOST features for program two } Output: { cm_return - if =0, success distance - Euclidean distance (can be more than 1) }
def module::change_work_dir | ( | i | ) |
Chaning working directory Input: { (work_dir) - change to this working directory before running code or (work_dir_repo_uoa) - change to the working directory in the repository (repo UOA) (work_dir_module_uoa) - change to the working directory in the repository (module UOA) (work_dir_data_uoa) - change to the working directory in the repository (data UOA) } Output: { cm_return - if =0, success work_dir - selected working directory }
Chaning working directory Input: { work_dir - if !='', go to this directory (work_dir_repo_uoa) - if !='', go to the directory for this cM entry (work_dir_module_uoa) - if !='', go to the directory for this cM entry (source_code_uoa) - if !='', go to the directory for this cM entry } Output: { cm_return - if =0, success work_dir - selected working directory }
def module::clean | ( | i | ) |
Search Input: { (index_repo_uoa) - Repository UOA to clean indexes (index_module_uoa) - Module UOA to clean indexes (index_data_Uoa) - Data UOA to clean indexes } Output: { cm_return - return code >0 if error }
Clean repository Input: { target_repo_uoa - target repo UOA (target_module_uoa) - clean data for this module (cm_classes_uoa) - prune data by classes (target_data_uoa) - data to clean } Output: { cm_return - return code >0 if error cm_path - path to file cm_uid - generated uid }
def module::cmd_prune_flags | ( | i | ) |
def module::compare_systems | ( | i | ) |
def module::compile_program | ( | i | ) |
def module::configure | ( | i | ) |
def module::convert_ct_multi_array_to_csv | ( | i | ) |
Convert cTuning multi-dimensional array to csv Input: { cm_multi_array_s - array of strings to convert ct_dimensions - list of dimensions to add ct_dim_desc - dimension description (categorical/continuous, etc) ct_output_file - output csv file (csv_separator) - separator for data in csv (';' by default) (csv_decimal_mark) - decimal char (default ',') (csv_no_header) - if 'yes', do not add header } Output: { cm_return - return code >0 if error }
def module::convert_ct_space_points_to_python_array | ( | i | ) |
Convert cTuning space points to python array Input: { ct_module_uoa - module to visualize ct_points - list of UOA with points ct_axes - list with flattened axes (if empty, all axes are taken) ct_axes_desc - axes description (use_multiple_graphs) - if 'on' separate entries into different sub-graphs (data_entry_to_sort) - if sort and multiple entries, use this entry number to sort (-1) (fix_additional_dimensions) - if 'yes', fix or range additional dimensions (fix_additional_dimensions_data) - dictionary of format: {"flattened key A":{'less':X, 'equal':Y, 'more':Z}, ...} } Output: { cm_return - return code >0 if error ct_array - multidimensional array ([ [],[],...]) ct_array_id - 1D array with {cm_uid, cm_uoa} ct_axes - either original one or ALL axes if original len==0 }
def module::convert_field_to_html | ( | i | ) |
Convert cM data field to html Input: { cm_separator - appendable separator to deal with json hierarchy cm_separator_form - appendable separator to deal with array hierarchy (and with additional forms) cm_key - array key cm_value - array value cm_data_desc - data description in special cM format cm_mode - add | update (cm_back_json) - array with previous cM state (cm_array_orig) - original array (not changing through the calls) (cm_cur_array) - current array to change if needed (cm_cur_key) - current key to change array if needed (cm_form_explore) - form explore parameters (cm_explore) - if 'yes', prepare exploration choices (cm_form_explore_color) - prefix for style (cm_explore_commands) - array with exploration commands (cm_explore_state) - {} array with choices and exploration state (cm_explore_level) - exploration level ('' - coarse-grain, '1' - next level, ...) (hide_add_new_object) - if 'yes', do not show 'add new object' (cm_disable_coloring) - if 'yes', disable coloring of dimensions, characteristics, properties, state, etc (cm_admin) - if 'yes', override access control - only internal use, can not be used during web access (explicitly removed) } Output: { cm_return - return code >0 if error cm_value_html - prepared html cm_value_text - text (in URL, etc) }
def module::convert_file_to_upload_string | ( | i | ) |
def module::convert_flags_from_ccc | ( | i | ) |
Convert flags from CCC framework Input: { cm_filename - CCC compiler configuration file (add_base_opt_upfront) - if 'yes', add base opts upfront (such as -O3, -O2, etc). One should edit final config! (create_cm_entry) - if 'yes' add to cM (repo_uoa) - repository to create entry (data_uoa) - data uoa to create entry }
def module::convert_points_to_flat_csv | ( | i | ) |
Convert points into flat CSV (possibility to choose dimensions). Needed for machine learning and data mining. Input: { ct_data_uoa - Entry with points (ct_data_selection) - Multiple entries if needed (ct_data_selection_from_files) - Multiple entries from json files prepared by this function (ct_module_uoa) - Specify module from scenarios (otherwise ctuning.space by default) (use_multiple_graphs) - if 'on' spread data from ct_data_selection into different dimensions (ct_dimensions) - List of dimensions to add (ct_repo_uoa) - Repository (ct_output_file_s_json) - output json file (values as strings) (ct_output_file_json) - output json file (real values) (ct_output_file_csv) - output csv file (csv_separator) - separator for data in csv (';' by default) (csv_decimal_mark) - decimal char (default ',') } Output: { cm_return - return code >0 if error ct_multi_array_s - list of dictionaries per multi-graph of format 'flat_dimension1':['value1', 'value2', ...], 'flat_dimension2':[...], ... ct_multi_array - list of dictionaries per multi-graph of format 'flat_dimension1':[value1, value2, ...], 'flat_dimension2':[...] ... where all values are based on cM dimensions description in a note ct_dim_desc - dictionary of format {'flat_dimension': cM dimension description (categorical/continuous/etc) } Note: cM dimensions description = { type - "text", "integer", "uoa", "float", etc data_desc - textual description of dimension has_choice - if 'yes', this dimension is categorical choice" - [list of choices] explore_start - if !='', dimension is continuous; min value explore_stop - max value explore_step - step explore_prefix - add prefix before dimension (useful for compiler flags with conintuous parameters) }
def module::convert_python_array_to_csv | ( | i | ) |
def module::convert_python_array_to_gws_array | ( | i | ) |
Convert python array to Google Web Services array Input: { ct_array - collective tuning array ct_array_id - array with space data points UIDs ct_axes - axes ct_axes_desc - axes description ct_axes_type - needed types for Google graph (convert_text_axes_to_float) - convert text axes to consequitive floats } Output: { cm_return - return code >0 if error s_array - GWS array s_array_id - GWS array ID (to process clicks) }
def module::convert_str_to_sha1 | ( | i | ) |
Convert string to sha1 Input: { cm_string - string (cm_console) - by default, do not output anything except errors if 'txt' then output as plain text if 'web' then output for web (html, image, etc) if 'json' then output as json } Output: { cm_return - return code >0 if not authentificated cm_string_sha1 - password in SHA1 (digest) (only if console is not json, otherwise doens't work) cm_string_sha1_hex - password in SHA1 (digest in hex) cm_string_sha1_base64 - BASE64 (SHA1 digest) - compatible with htpasswd format }
def module::convert_text_axes_to_float | ( | i | ) |
Finish python graph and save to file Input: { ct_array - collective tuning array ct_axes - axes ct_axes_desc - axes description (num_axes) - number of axes to process (to allow to convert only X sometimes) } Output: { cm_return - return code >0 if error ct_array - updated ct array unique - if converted from string to number, unique values unique_string - if converted from string to number, unique values converted to string }
def module::copy | ( | i | ) |
Copy repository to another one (and clean target if needed) Input: { orig_repo_uoa - original repository (orig_module_uoa) - copy data for this module (cm_classes_uoa) - prune data by classes target_repo_uoa - target repository (clean_target_repo) - force cleaning of target repo } Output: { cm_return - return code >0 if error cm_path - path to file cm_uid - generated uid }
def module::copy_data | ( | i | ) |
Copy data entry Input: { (orig_repo_uoa) - repo UOA of original data orig_module_uoa - module of the data to copy orig_data_uoa - original data UOA (target_repo_uoa) - target repo where to copy data (if =='', repo of the original data is used) (target_data_uoa) - target data UOA (target_data_uid) - target data UID (if not set, original is used) (remove_alias) - if 'yes', remove alias (generate_new_uid) - if 'yes', generate new target UID (overwrite) - if 'yes' and target exists, overwrite (move) - if 'yes', delete original after copy (cm_array_update) - update data of created entry } Output: { cm_return - return code = 0, if successful }
def module::create | ( | i | ) |
def module::create_button | ( | i | ) |
def module::create_button_back | ( | i | ) |
def module::default | ( | i | ) |
def module::delete_data | ( | i | ) |
Delete data Input: { cm_run_module_uoa | cm_module_uoa - module UOA cm_data_uoa - data UOA (cm_repo_uoa) - repo UOA (cm_force_delete) - if 'yes', force delete (cm_console) - by default, do not output anything except errors if 'txt' then output as plain text if 'web' then output for web (html, image, etc) if 'json' then output as json (cm_admin) - if 'yes', override access control - only internal use, can not be used during web access (explicitly removed) } Output: { cm_return - return code >0 if error -1 if entry exists }
def module::delete_data_through_web | ( | i | ) |
def module::detect | ( | i | ) |
Detect system properties Input: { (host_os_uoa) - host OS on which cM is running to detect which scripts to use for detection. If not set, the default one is taken from cM configuration. (target_os_uoa) - target OS to detect. If =='', host OS is used (get_uoa_from_repo) - if 'yes', check if already in repo and return UOA (detect_repo_uoa) - search entries in this repository (if not set, do not search), if not found, add entries (record_if_doesnt_exist) - if 'yes', record in repo if doesn't exist (ask_for_strings_if_not_detected) - if 'yes' and strings are not detected, ask for them (add_all) - if 'yes' add all information including generated files about system (used in private setups to keep all information for debugging or machine learning (to improve models and find appropriate features)) } Output: { cm_return - return code = 0, if successful cm_array - { processor_string system_string available_physical_memory } processor_string - processor os_string - OS system_string - System memory_string - All memory all_frequencies_string - All possible frequencies processor_uoa - detected processor UOA or taken from the input os_uoa - detected OS UOA or taken from the input system_uoa - prepared system UOA }
def module::detect_form_params | ( | i | ) |
Detect various parameters (commands) in cM web forms Input: { cm_array - array of form #prefix#var or #prefix@command cm_prefix - prefix } Output: { cm_return - return code >0 if error cm_array - parsed array with keys cm_commands - parsed array with commands cm_explore - parsed array with exploration dimensions }
def module::detect_host_family | ( | i | ) |
Detect main OS version using python Input: { (cm_repo_uoa) - which repo to use to get info (to support remote ones) } Output: { cm_return - return code = 0, if successful family - OS family cm_os_uoa_list - list of OSes from the same family (through search) cm_host_uoa - most close OS UOA from known cm_host_alias - alias for cm_host_uoa }
def module::detect_representative_points | ( | i | ) |
Detect representative points to build the same model Input: { model_type - "regression" or "classification" data ct_dimensions_input (ct_dimensions_output) - for comparison desc - cM data description (max_variation_percent) - for comparison, report points where variation is more than this number (default=0.2) } Output: { cm_return - return code >0 if error results - list with info about models including model_file_size model_rmse }
def module::download | ( | i | ) |
Download repository using cM web interface (possibly substitute old entries) Input: { (web_repo_uoa) - repository UOA; if empty, update all repositories that are shared (overwrite) - if 'yes', overwrite entries (download_all) - if 'yes', download all (skip_large_download) - if 'yes', skip entries with large downloads (interactive) - if 'yes', interactive (ask) - if 'yes', ask what to do with modules with many entries (cm_remote_user_uoa) - change default user to access repo ... } Output: { cm_return - return code >0 if error }
Download data entry Input: { cm_run_module_uoa | cm_module_uoa - module UOA cm_data_uoa - data UOA (cm_repo_uoa) - repo UOA cm_all_files - if 'yes', download .cm description files this is needed for the future cloning of repositories to avoid using all the time SVN/GIT for simple sharing ... } Output: { cm_return - return code >0 if error ... - various parameters to set http headers, etc }
Definition at line 3803 of file module.py.
Referenced by org::openme::openme::remote_access().
def module::explore | ( | i | ) |
Explore choices Input: { cm_choices - TODO: describe format, see ct-compiler gcc-4.6.2 cm_exploration_strategy - random (choice_rnd_seed) - random seed *concatenate_output) - if 'yes', concatenate list to string (output_key) - use this value to set output key instead of cm_choice. Mainly used for command line exploration. } Output: { cm_return - if =0, success cm_choice or value of output_key - list of selected choices }
Explore pipeline choices Input: { To update ... exploration_scenario - exploration scenario for this pipeline exploration_scenario_module_uoa - exploration scenario module UOA number_of_iterations - number of coarse grain iterations skip_if_entry_exists - if 'yes' and coarse-grain entry exists, skip it useful for exploring only new coarse-grain points in space state - init state desc - state description (mobile_mode) - 'build' or 'process' - mode during crowdsourcing auto-tuning using mobiles (copy_built_executables) - if 'yes', copy all built executables to a_<UID>.out (copy_src_to_tmp) - if 'yes', copy sources to a tmp repo (useful for parallel cloud/mobile tuning) (system_info) - detected system info by the mobile/cloud/grid system (new_src_uoa) - new src UOA in tmp repository (measured_frequency) - measured frequency (if supported) (execution_info) - list of list of execution times per binary - 1 coarse-grain exploration step (execution_info_min) - list of minimal execution times per 1 coarse-grain exploration step (execution_info_ct_repeat_main) - CT_REPEAT_MAIN got during calibration (built_files) - list of unique built files (with UIDs) } Output: { cm_return - if =0, success (additional parameters from the last scenario) (built_files) =[{"name","size","prepared_compiler_flags"},...] (new_src_uoa) - if src were copied, new UID in tmp repo }
Explore a few points from mobile, cloud, etc Input: { predefined_uoa - data UOA with pre-defined input to compiler optimization scenario mobile_mode - 'build' or 'process' - mode during crowdsourcing auto-tuning using mobiles (system_info) - detected system info by the mobile/cloud/grid system (new_src_uoa) - new src UOA in tmp repository (measured_frequency) - measured frequency (if supported) (execution_info) - list of list of execution times per binary - 1 coarse-grain exploration step (execution_info_min) - list of minimal execution times per 1 coarse-grain exploration step (execution_info_ct_repeat_main) - CT_REPEAT_MAIN got during calibration (built_files) - list of unique built files (with UIDs) } Output: { cm_return - if =0, success (new_src_uoa) - if src were copied, new UID in tmp repo }
def module::filter_special_str | ( | i | ) |
def module::find_aggregated_entry | ( | i | ) |
def module::find_cid | ( | i | ) |
def module::find_data_without_access_control | ( | i | ) |
Test some funcionality before moving to a proper test Input: { (prune_repo_uoa) - prune search by a given repository (prune_module_uoa) - prune search by a given module (prune_data_uoa) - prune search by a given data } Output: { cm_return - return code >0 if error }
def module::find_key_value_in_array | ( | i | ) |
Find key/value pair in array Input: { cm_array - array cm_search - {"key": key, "value": value} or {"value":value} (use_flat_array) - if 'yes', keys should be in "flat array" format (##x#y ...) (strict_match) - if 'yes', make exact comparison of values } Output: { cm_return - return code >0 if error cm_found - True if found }
def module::find_path_to_data | ( | i | ) |
def module::find_path_to_repository | ( | i | ) |
Find path to a given repository Input: { (find_repo_uoa) - repo UOA (get_default_repo) - if 'yes' and find_repo_uoa=='', get path to default repo (cm_console) - if 'txt', output as text to console (cm_admin) - if 'yes', override access control - only internal use, can not be used during web access (explicitly removed) } Output: { cm_return - return code >0 if error cm_path - path to repository cm_data_obj - cM data object cm_uid - uid (from UOA) cm_alias - alias (from UOA) cm_uoa - cm_data_uoa (alias or uid if alias=='') (cur_module_uoa) - current module UOA if in current path (cur_data_uoa) - current data UOA if in current path }
def module::finish_python_graph | ( | i | ) |
Finish python graph and save to file Input: { ct_graph - graph type matplot_obj - prepared MatPlot object cm_file_ext - file extension (url) - starting url of cM (reproduce) - parameters to reproduce/save graph (save_graph) - prepare for saving (cm_console) - if 'txt' output path to the generated image to the console (legend) - if !='', add to the end of the graph (mpl_extra_params) - if python mat plot is used, this dictionary has extra params mpl_image_size_x - image X size in inch mpl_image_size_y - image Y size in inch mpl_image_dpi - image DPI } Output: { cm_return - return code >0 if error }
def module::fit | ( | i | ) |
Fit data using existing models (combine build and predict) Input: { data ct_dimensions_input (ct_dimensions_output) - for comparison desc - cM data description (max_variation_percent) - for comparison, report points where variation is more than this number (default=0.2) } Output: { cm_return - return code >0 if error results - list with info about models including model_file_size model_rmse }
def module::flush | ( | i | ) |
def module::functionality | ( | i | ) |
def module::gen_cm_tmp_file | ( | i | ) |
def module::gen_docs | ( | i | ) |
def module::gen_stats | ( | i | ) |
def module::gen_uid | ( | i | ) |
Generate cM UID Input: { (cm_console) - by default, do not output anything except errors if 'txt' then output as plain text if 'web' then output for web (html, image, etc) if 'json' then output as json } Output: { cm_return - return code = 0, if successful cm_uid - cM UID }
def module::generate | ( | i | ) |
Generating research topics and papers for "low-hanging fruits" ;) Input: { cm_data_uoa - data with topic template (cm_repo_uoa) - repository } Output: { cm_return - return code >0 if error }
Generate qr-code Input: { cm_string - string to convert to qr-code qr_level - qr_level (4,5,etc) image_size - picture size } Output: { cm_return - return code >0 if error cm_full_filename - file with image }
def module::get_all_repo_paths | ( | i | ) |
Get paths for all repositories (has to be low level operation!) Input: { (cm_admin) - if 'yes', override access control - only internal use, can not be used during web access (explicitly removed) } Output: { cm_return - return code >0 if error cm_array - array with paths }
def module::get_env | ( | i | ) |
def module::get_info_about_current_path | ( | i | ) |
Get info about current path Input: {} Output: { cm_return - return code >0 if error -1 if doesn't exist cm_path - path to the local repository (cm_module_uoa) - module UOA of the current path (cm_data_uoa) - data UOA of the current path (cm_repo_uoa) - repo UOA (cm_repo_uid) - repo UID }
def module::get_info_about_full_hosts | ( | i | ) |
def module::get_list_of_all_files | ( | i | ) |
Get a list of all files in a given directory recursively Input: { cm_module_uoa - module UOA cm_data_uoa - data UOA (cm_repo_uoa) - repo UOA cm_path_ext - path extension limit - limit number of files (if huge directories) number - cur. number of files cm_all_files - if 'yes' do not ignore special files (like .cm) } Output: { cm_return - return code = 0 if successful cm_array - list of files }
def module::get_state | ( | i | ) |
Get current state Input: { (host_os_uoa) - host OS on which cM is running to detect which scripts to use for detection. If not set, the default one is taken from cM configuration. (target_os_uoa) - target OS to detect. If =='', host OS is used } Output: { cm_return - return code = 0, if successful system_state - [windows] [unix] }
def module::get_stats_for_benchmarks | ( | i | ) |
Compare systems through benchmarking Input: { (data) - list of data entries to analyze (prune_by_summary_point_relaxed) - if data is not provided, search will be performed for all entries; returned entries can be pruned by this dictionary using 'summary_point_relaxed' in entries characteristic_key - key to get characteristic (will be converted to float) (cm_repo_uoa) - working repository } Output: { cm_return - if =0, success stats - dictionary of format { "benchmark1":{ "min" - min value "max" - max value "delta" - max-min "percent_delta_to_min" - delta/min "mean" - mean "std" - standard deviation "percent_std_to_min" - standard deviation/min } ... } }
def module::gws_2d_column_chart | ( | i | ) |
Google Web Services 2D Column Chart Input: { ct_module_uoa - module to visualize ct_array - collective tuning array ct_array_id - array with space data points UIDs ct_axes - axes ct_axes_desc - axes description ct_points - selected entries url - starting url of cM (convert_text_axes_to_float) - convert text axes to consequitive floats (legend) - if !='', add to the end of the graph } Output: { cm_return - return code >0 if error }
def module::gws_2d_scatter_chart | ( | i | ) |
Google Web Services 2D Scatter Chart Input: { ct_module_uoa - module to visualize ct_array - collective tuning array ct_array_id - array with space data points UIDs ct_axes - axes ct_axes_desc - axes description ct_points - selected entries url - starting url of cM (convert_text_axes_to_float) - convert text axes to consequitive floats (legend) - if !='', add to the end of the graph } Output: { cm_return - return code >0 if error }
def module::help | ( | i | ) |
def module::index | ( | i | ) |
Add or delete index Input: { cm_module_uoa - module UOA (cm_module_uid) cm_data_uoa - data UOA (cm_data_uid) (cm_repo_uoa) - repo UOA (cm_repo_uid) mode - 'delete' | 'add' (cm_index_uoa) - index configuration (sequential) - if 'yes' perform sequential index (very slow, but reports errors) } Output: { cm_return - return code >0 if error }
Show cM main web page Input: { (cm_web_preprocess) - if 'yes', preprocess request (usually user authentication) (cm_menu) - select user menu } Output: { cm_return - return code >0 if error }
def module::index_seq | ( | url, | ||
jindex, | ||||
mode, | ||||
module_uid, | ||||
data_uid | ||||
) |
def module::info | ( | i | ) |
def module::init | ( | i | ) |
def module::install | ( | i | ) |
Install code and set environment Input: { (install_data_uid) - create an entry with this UID (install_data_alias) - create an entry with this alias (install_data_display_as_alias) - use this display as alias for a generated entry (install_module_uoa) - use this module to create an entry (if not set, use 'code') (install_repo_uoa) - use this repo to create an entry (cm_array) - add this data to the new entry target_os_uoa - target os to install script (add_rem_to_script) - add rem to install script } Output: { cm_return - return code = 0, if successful Parameters from entry creation script_filename - script filename script_name - full script name }
Install cM package Input: { package_data_uoa - UOA of package to install (package_repo_uoa) - repo of the package package_host_os_uoa - UOA of host OS (to set up script execution) (install_data_uid) - UID of the code where package will be installed if install_data_uid=='' it will be randomly generated (install_data_alias) - use this alias for a generated entry (install_data_display_as_alias) - use this display as alias for a generated entry If package should be built: build_target_os_uoa - target OS uoa for building (code_deps) - list with code UOA for dependencies [{"index":"uoa"} ...] (run_target_processor_uoa) - target processor UOA (not strictly needed - can add some helper parameters before executing code) (run_set_env2) - array with environment variables to be set before executable (add_rem_to_script) - add rem to install script (add_to_code_entry) - add array to code entry (skip_extract_and_build) - if 'yes', skip copy, extract and build (by setting CM_SKIP_BUILD=yes) } Output: { cm_return - return code = 0, if successful Output from 'code install' }
def module::installed | ( | i | ) |
List installed packages Input: { (code_repo_uoa) - repo UOA to list code (prune_by_class_uoa) - prune by 1 class UOA (prune_by_name) - prune by name (checks for inclusion; case insensitive) (fuzzy_match) - if 'yes', check inclusion of 'prune_by_name' in name, otherwise exact match (prune_by_name_uoa) - prune by name UOA (prune_by_type) - prune by "library_or_plugin" or "package" (prune_by_host_os) - prune by host os (prune_by_target_os) - prune by target os (prune_by_target_processor) - prune by target processor (prune_by_compiler) - if library or plugin, prune by compiler (prune_by_os_from_kernel) - if 'yes', take OS from kernel (only_uoa) - return only UOA } Output: { cm_return - return code = 0, if successful final - list with code entries in cM 'list' format }
def module::list_data | ( | i | ) |
List data Input: { cm_run_module_uoa | cm_module_uoa - module UOA (cm_repo_uoa) - repo UOA (cm_data_uoa) - data UOA (cm_console) - by default, do not output anything except errors if 'txt' then output as plain text if 'web' then output for web (html, image, etc) if 'json' then output as json (cm_only_uid) - 'yes' to show only UID (cm_class_uoa) - prune by 1 class UOA or (cm_classes_uoa) - prune by classes UOA (list) (cm_only_available_modules) - if 'yes', return only available modules in a given repository (skip_data_cfg) - if 'yes', do not add data cfg (show_alias_and_uid) - if 'yes', show alias (UID) (show_display_as_alias) - if 'yes', show in format DISPLAY_AS_ALIAS (UID / ALIAS) } Output: { cm_return - return code >0 if error -1 if entry exists cm_array - array with UOA cm_mixed - list of {cm_uid, cm_alias, cm_uoa, cm_display_as_alias, cm_display_html} }
def module::list_files_through_web | ( | i | ) |
def module::load_data | ( | i | ) |
Load data Input: { cm_module_uoa - module UOA cm_data_uoa - data UOA (cm_repo_uoa) - repo UOA (cm_console) - by default, do not output anything except errors if 'txt' then output as plain text if 'web' then output for web (html, image, etc) if 'json' then output as json (cm_web) - yes, for web environment (lock_acquire) - if 'yes', acquire lock (lock_expire) - for how long to acquire lock (secs) (cm_admin) - if 'yes', override access control - only internal use, can not be used during web access (explicitly removed) } Output: { cm_return - return code = 0, if successful < 0, if error 32 - lock acquired by someone else (cm_error) - error text if return code > 0 cm_data_obj - cM data object cm_path - path to data entry cm_path_module - path to module entry with this data cm_uid - uid (from UOA) cm_alias - alias (from UOA) cm_display_as_alias - taken from the data cm_repo_uoa - UOA of a repository cm_repo_uid - UID of a repository cm_display_html - prepare display: "display as alias" or <I>alias</I> or UID (lock_uid) - UID of the aquired lock }
def module::load_default_kernel_data | ( | i | ) |
def module::login | ( | i | ) |
def module::logout | ( | i | ) |
def module::mobile_node_prepare | ( | i | ) |
def module::mpl2_2d_scatter_chart | ( | i | ) |
def module::mpl2_2d_scatter_chart_a | ( | i | ) |
def module::mpl2_2d_warm_graph | ( | i | ) |
def module::mpl2_2d_warm_graph_a | ( | i | ) |
def module::mpl_1d_density_graph | ( | i | ) |
Python Matplot 1D density graph Input: { ct_array - collective tuning array ct_array_id - array with space data points UIDs ct_axes - axes ct_axes_desc - axes description ct_points - selected entries reproduce - parameters to reproduce/save graph (url) - starting url of cM (save_graph) - if 'yes', save graph (cm_file_ext) - file extension if save_graph=='yes' (convert_text_axes_to_float) - convert text axes to consequitive floats (mpl_extra_params) - if python mat plot is used, this dictionary has extra params mpl_image_size_x - image X size in inch mpl_image_size_y - image Y size in inch mpl_image_dpi - image DPI mpl_detach_graph - if 'on', detach graph } Output: { cm_return - return code >0 if error }
def module::mpl_2d_column_chart | ( | i | ) |
Python Matplot 2D column chart Input: { ct_array - collective tuning array ct_array_id - array with space data points UIDs ct_axes - axes ct_axes_desc - axes description ct_points - selected entries reproduce - parameters to reproduce/save graph (url) - starting url of cM (save_graph) - if 'yes', save graph (cm_file_ext) - file extension if save_graph=='yes' (convert_text_axes_to_float) - convert text axes to consequitive floats (mpl_extra_params) - if python mat plot is used, this dictionary has extra params mpl_image_size_x - image X size in inch mpl_image_size_y - image Y size in inch mpl_image_dpi - image DPI mpl_detach_graph - if 'on', detach graph } Output: { cm_return - return code >0 if error }
def module::mpl_2d_scatter_chart | ( | i | ) |
Python Matplot 2D scatter chart Input: { ct_array - collective tuning array ct_array_id - array with space data points UIDs ct_axes - axes ct_axes_desc - axes description ct_points - selected entries reproduce - parameters to reproduce/save graph (url) - starting url of cM (save_graph) - if 'yes', save graph (cm_file_ext) - file extension if save_graph=='yes' (convert_text_axes_to_float) - convert text axes to consequitive floats (mpl_extra_params) - if python mat plot is used, this dictionary has extra params mpl_image_size_x - image X size in inch mpl_image_size_y - image Y size in inch mpl_image_dpi - image DPI mpl_detach_graph - if 'on', detach graph } Output: { cm_return - return code >0 if error }
def module::mpl_get_available_fonts | ( | i | ) |
def module::native_run | ( | i | ) |
def module::output_known_characteristics | ( | i | ) |
def module::postprocess_web_request | ( | i | ) |
Postprocess web request Input: { (cm_web_session) - $_SESSION (cm_web_get) - $_GET (cm_web_post) - $_POST (cm_web_cookie) - cookies (cm_user_uoa) - user UOA (cm_user_password) - user password in plain text (not secure, usually used for web form authentication) (cm_user_password1) - user password in SHA1 (more secure, can be stored in environment, session or cookies) cM authenicates by taking SHA1(cm_user_password1)... (cm_web_module_uoa) - module to envoke (cm_web_action) - action for the module } Output: { cm_return - return code >0 if error }
def module::predict | ( | i | ) |
Predict using some model Input: { model_module_uoa - model's module (model_data_uoa) - model's data (if need to record) model_name - (earth, svm) data ct_dimensions_input (ct_dimensions_output) - for comparison desc - cM data description (max_variation_percent) - for comparison, report points where variation is more than this number (default=0.2) } Output: { cm_return - return code >0 if error (rmse) - if comparison, root mean square error for predictions vs original (max_var) - list of points with variation more than max_variation_percent }
Predict using model Input: { model_file data ct_dimensions_input (ct_dimensions_output) - for comparison desc - cM data description model_name - (earth, svm) (max_variation_percent) - for comparison, report points where variation is more than this number (default=0.2) } Output: { cm_return - return code >0 if error (rmse) - if comparison, root mean square error for predictions vs original (max_var) - list of points with variation more than max_variation_percent }
def module::predict_opts | ( | i | ) |
Predict optimizations Input: { In non-web mode: (program_milepost_properties) - if not empty, return most close programs (cm_submit_prune) - if 'yes', prune results to find most close program (opt_predictor) - 'nearest_neighbour' for nearest neighbour classifier (cm_repo_uoa) - repo to prune results (pruning_similar_programs) - direct pruning parameters (number_of_returned_items) - number of returned items } Output: { cm_return - if =0, success }
def module::prepare_choices | ( | i | ) |
Prepare tuning dimension Input: { cm_array - pipeline array cm_data_desc - pipeline data description cm_explore - exploration parameters cm_explore_state - exploration state cm_explore_level - exploration level ('' - coarse-grain, '1' - next level, ...) random_seed - random seed } Output: { cm_return - if =0, success cm_explore_state - exploration state cm_exploration_finished - if 'yes', exploration finished (for nested loops) }
def module::prepare_cmd | ( | i | ) |
Prepare command line for the code Input: { (code_module_uoa) - module of the working data (code_data_uoa) - working data uoa (run_cmd_key) - name if multiple CMDs available for this program (dataset_uoa) - UOA of dataset used to prepare this cmd (os_uoa) - OS UOA (compilation_type) - static or dynamic } Output: { cm_return - if =0, success (cm_error) - if cm_return>0, error text run_time={ run_env - set environment before command line - the following params will be aggregated to 1 line run_cmd1 run_cmd_binary_name run_cmd2 run_cmd_main run_cmd3 run_cmd_out1 run_cmd_out2 run_output_files run_input_files dataset_files } build_compiler_vars - possible compiler variables (often set hardwired dataset) }
def module::prepare_content_header | ( | i | ) |
def module::prepare_env_for_all_codes | ( | i | ) |
Prepare environment for all codes Input: { code_deps - list with code UOA for dependencies [{"index":"uoa"} ...] os_uoa - OS UOA no_strict_check - if 'yes', do not check if code dependency was installed properly } Output: { cm_return - return code = 0, if successful cm_string - environment string cm_array - array with environment setting (for scripts) include_paths1 - list with paths with include directories lib_paths - list with libraries env_separator - environment separator (just in case to avoid double loading of OS) }
def module::prepare_env_vars | ( | i | ) |
Prepare environment variables Input: { array - array of environment variables to set prefix - add prefix to set environment variable separator - add separator for the command line (quotes) - use quotes? } Output: { cm_return - if =0, success cm_string - cmd with prepared environment (export A=B; export C=D; ...) cm_string1 - simplified string (CM_ALL_ENV="A=B C=D E=F") cm_array - array with environment setting (for scripts) }
def module::prepare_environment | ( | i | ) |
def module::prepare_flags | ( | i | ) |
Prepare flags from array Input: { cm_data_uoa - data with compiler description all_flags - {} all flags (all_flags_order) - [] order of flags } Output: { cm_return - if =0, success prepared_flags - string with prepared flags all_flags_order - [] order of flags (if original was empty, prepare order according to sort_index in description) }
def module::prepare_input_params | ( | i | ) |
Prepare input parameters for this scenario Input: { cm_form_array - array with forms form_exists - yes or no cm_array1 - array from form 1 (coarse-grain points) cm_array2 - array from form 2 (additional parameters) just in case caller_cfg - caller module cfg } Output: { cm_return - if =0, success }
def module::prepare_path | ( | i | ) |
def module::prepare_script | ( | i | ) |
Prepare script with code deps Input: { script_name - script name (usually without extensions) target_os_uoa - target os to get script extension and other parameters (skip_extension) - if 'yes', script already has extension (cm_path) - path to create script (set_env1) - add environment vars (1) to script (code_deps) - add code deps to script (set_env2) - add environment vars (2) to script (run_commands_before) - add some commands before main command (run_cmd) - add main command (run_commands_after) - add some commands after main command } Output: { cm_return - return code = 0, if successful cm_path - full name of the created script with path }
def module::prepare_sub_script | ( | i | ) |
Prepare sub_script with extension Input: { run_script - script name (usually without extensions) (run_script_uoa) - script UOA if any target_os_cfg - target os cfg to get script extension and other parameters } Output: { cm_return - return code = 0, if successful cm_path - full name of the created script with path }
def module::prepare_summary_point | ( | i | ) |
Prepare summary point (to save a unique experiment in a repository) Input: { state - pipeline state to use dimensions for search } Output: { cm_return - if =0, success summary_point1 - summary point (dimensions) - strict summary point to detect the same experiments summary_point2 - summary point (dimensions) - relaxed summary point for viewing and pruning }
def module::prepare_tuning_dimension_for_web | ( | i | ) |
def module::preprocess_web_request | ( | i | ) |
Preprocess web request (convert $_GET and $_POST into one array and envoke cM module) Input: { (cm_web_session) - $_SESSION (cm_web_get) - $_GET (cm_web_post) - $_POST (cm_web_cookie) - cookies (cm_user_uoa) - user UOA (cm_user_password) - user password in plain text (not secure, usually used for web form authentication) (cm_user_password1) - user password in SHA1 (more secure, can be stored in environment, session or cookies) cM authenicates by taking SHA1(cm_user_password1)... (cm_web_module_uoa) - module to envoke (cm_web_action) - action for the module } Output: { cm_return - return code >0 if error ... - output from the envoked module }
def module::print_help | ( | i | ) |
def module::process | ( | i | ) |
Run collective tuning pipeline Input: { space - available points in space pipeline_input - pipeline input pipeline_output - pipeline output desc - state description tune_characteristics - list of characteristics to tune (priorities, such as global execution time; code size; energy; compilation time, etc - depending on user scenario) calcluate_improvement_over_ref - if !='', use this as the key in space to calcluate improvements over first point max_improvement_threshold - if at least one of the max improvement more than threshold, add } Output: { cm_return - if =0, success state - final state space - (updated) space must_add - if 'yes', should add this point }
Run collective tuning pipeline Input: { state - init state (state1) - state where to record characteristics - if empty, use 'state' desc - state description } Output: { cm_return - if =0, success state - final state state1 - state1 }
Analyze variation of variables Input: { array - dictionary of format { "varA":[valueA1, valueA2, ...], "varB":[valueB1, valueB2, ...], ... } } Output: { cm_return - if =0, success array - dictionary of format { "varA":{ "min" - min value "max" - max value "delta" - max-min "percent_delta_to_min" - delta/min "mean" - mean "std" - standard deviation "percent_std_to_min" - standard deviation/min } ... } }
def module::process_web_request | ( | ix | ) |
def module::prune | ( | i | ) |
def module::record | ( | i | ) |
def module::register_func | ( | i | ) |
Register function (action) in a module Input: { cm_run_module_uoa - module UOA (cm_common_func) - if ='true', then the module is called from some other module (new_func) - new function name (new_desc) - new desc (new_index) - new index (if empty, use func) } Output: { cm_return - 0 }
def module::rename_data | ( | i | ) |
Rename data entry Input: { (orig_repo_uoa) - repo of original data orig_module_uoa - module of original data orig_data_uoa - data UOA to rename new_data_alias - new data alias (leave empty to remove alias) new_data_uid - new data UID (leave empty to keep old one) } Output: { cm_return - return code = 0, if successful }
def module::repair | ( | i | ) |
def module::reproduce | ( | i | ) |
Reproduce collective tuning pipeline state Input: { reproduce_module_uoa - pipeline state module UOA reproduce_data_uoa - pipeline state data UOA (reproduce_repo_uoa) - pipeline state repo UOA (point) - point in state (by default 0) data_uoa - use this data UOA to load recorded state (repo_uoa) - use this repo UOA to load recorded state (pipeline_uoa) - entry with pipeline or (pipeline) - pipeline array [{cm_access input}] (record_in_space) - if 'yes', record in cTuning space (record_repo_uoa) - if record, which repo to use (record_data_uoa) - if record, set data_uoa (if not set, generate randomly) } Output: { cm_return - if =0, success pipeline_input - input pipeline (original) pipeline_output - output pipeline (new) }
def module::restore_writing_in_kernel | ( | i | ) |
def module::run | ( | i | ) |
Run code (binary or script) Input: { (run_host_os_uoa) - host OS UOA (if not set, use default OS from kernel) (run_target_os_uoa) - target OS UOA (if not set, use run_host_uoa_os) (run_target_processor_uoa) - target processor UOA (not strictly needed - can add some helper parameters before executing code) -------------------------------------------------------------------------------------------- run_cmd_key - if !='', set all further run parameters (including target name) from the work_dir_data_uoa (must be set) (dataset_uoa) - use dataset to update cmd params from above or run_cmd - command line to run or run_script - use the following script (add extension from run_target_os) run_script_uoa - take script from this entry in os.script (add extension from run_target_os) or (run_cmd1) - following format: run_cmd1 run_cmd_name|binary_name run_cmd2 run_cmd_main run_cmd3 > run_cmd_out1 2> run_cmd_out2 (run_cmd_name) (run_cmd2) (run_cmd_main) or (binary_name) (run_cmd3) (run_cmd_out1) (run_cmd_out2) -------------------------------------------------------------------------------------------- (work_dir) - change to this working directory before running code or (work_dir_repo_uoa) - change to the working directory in the repository (repo UOA) (work_dir_module_uoa) - change to the working directory in the repository (module UOA) (work_dir_data_uoa) - change to the working directory in the repository (data UOA) (run_input_files) - list of files needed to run code (can be copied to the remote host system) (run_output_files) - list of files that will be created (for validation, etc) (code_deps) - list with code UOA for dependencies [{"index":"uoa"} ...] (skip_code_deps) - if 'yes', skip code deps (run_set_env1) - array with environment variables to be set before code deps (run_set_env2) - array with environment variables to be set before executable (run_commands_before) - run commands before executable (run_commands_after) - run commands after executable (cm_dependencies) - dependencies that set all other dependencies (code/include/lib). Format [{"class UOA":"code UOA"},{},...] (run_timeout) - time out for run (overheads may occur) (keep_all_files) - if 'yes', do not remove tmp files and do not clean up remote device (cm_verbose) - if 'yes', print all info } Output: { cm_return - return code = 0, if successful timed_out - 'yes' if timed out failed_run - 'yes' if return code !=0 or timed out or binary doesn't exist run_time_by_module - run time measured by module exit_code - exit code of the system call run_cmd - final run cmd work_dir - work directory run_output_files - final output files }
Run collective tuning pipeline Input: { state - init state Some additional debug/developer parameters cm_show_json - if 'yes', show input json (normally used before pipeline) cm_save_json_to_file - if !='', save input json to file (normally used before pipeline) cm_show_output_json - if 'yes', show final output state json on console cm_save_output_json_to_file - if !='', save final output state json to file } Output: { cm_return - if =0, success state - final state fail - True, if pipeline fail }
def module::run_stat | ( | i | ) |
Run collective tuning pipeline with statistical analysis of variability Input: { state - init state desc - pipeline description (needed to know characteristics) additional_params - scenario parameters dict "repeat_for_statistics" - number of repetitions for statistical analysis } Output: { cm_return - if =0, success state - final state }
def module::run_stat_cmd | ( | i | ) |
Run pipeline from CMD with user-friendly features Input: { (host_os_uoa) - if not selected, take from kernel (target_os_uoa) - if not selected, take from kernel (target_processor_uoa) - if not selected, use generic-x86-64 compiler_code_uoa or compiler_package_uoa or compiler_package_name (compiler_fuzzy_match) - if 'compiler_package_name' is used, and 'yes' check for inclusion rather than exact match of names program_uoa6 run_cmd_key dataset_uoa compile_script (kernel_repeat) (repeat_for_statistics) - number of pipeline repetitions (for statistical analysis) (statistics_module) - module for statistical analysis (update_state) - directly update state (only_prepare_state) - if 'yes', do not run pipeline but only update state and description } Output: { cm_return - if =0, success }
def module::scenario | ( | i | ) |
Build and run program, save to repository Input: { caller_module_uid - caller module UID state - pipeline state exploration_scenario - exploraiton scenario parameters summary_point - summary dimensions describing aggregated point aggregated_entry - all aggregated points space - if aggregated, current space original_input - to get other additional forms per scenario cm_data_desc - data description (from the original form) cm_form_explore - form explore (form the original form) number_of_iterations - max number of iterations } Output: { cm_return - if =0, success }
Build and run program, save to repository Input: { caller_module_uid - caller module UID state - pipeline state exploration_scenario - exploration scenario parameters work_repo_uoa - repository to save results summary_point - summary dimensions describing aggregated point aggregated_entry - all aggregated points space - if aggregated, current space original_input - to get other additional forms per scenario cm_data_desc - data description (from the original form) cm_form_explore - form explore (form the original form) number_of_iterations - max number of iterations mobile_mode - 'build' or 'process' - mode during crowdsourcing auto-tuning using mobiles copy_built_executables - if 'yes', copy all built executables to a_<UID>.out, state to a_<UID>.json and space to ct_space.json (new_src_uoa) - new src UOA in tmp repository (new_src_path) - new src path (measured_frequency) - measured frequency (if supported) (execution_info) - list of list of execution times per binary - 1 coarse-grain exploration step (execution_info_min) - list of minimal execution times per 1 coarse-grain exploration step (execution_info_ct_repeat_main) - CT_REPEAT_MAIN got during calibration (built_files) - list of unique built files (with UIDs) } Output: { cm_return - if =0, success }
Extract program MILEPOST static properties (features), save to repository Input: { caller_module_uid - caller module UID state - pipeline state exploration_scenario - exploraiton scenario parameters summary_point - summary dimensions describing aggregated point aggregated_entry - all aggregated points space - if aggregated, current space original_input - to get other additional forms per scenario cm_data_desc - data description (from the original form) cm_form_explore - form explore (form the original form) number_of_iterations - max number of iterations } Output: { cm_return - if =0, success }
def module::scenarios | ( | i | ) |
def module::search | ( | i | ) |
Search Input: { (repo_selection) - list of repo UOA to prune search (module_selection) - list of module UOA to prune search (key_X) - key to search (if use_flat_array=='yes', key should be in "flat array" format) (value_X) - value to search (find this occurance in the string) (cm_es_string) - ElasticSearch string (use_internal_search) - if 'yes', do not use ElasticSearch (use_flat_array) - if 'yes', keys should be in "flat array" format ( flat array is needed to access directly any key in dictionary hierarchy flat array format examples: {"x":{"y":"z"}} - "##x#y"="z" {"x":["y","z"]} - "##x@0"="y", "##x@1"="z" ) (show_search_progress) - if 'yes', show search progress (timeout) - timeout in seconds for internal slow search (strict_match) - if 'yes', use strict match (in internal search) } Output: { cm_return - return code >0 if error all - list with description of entries returned by search [ { cm_repo_uoa - Found entry's repository UOA (UID or alias) cm_repo_uid - Found entry's repository UID cm_module_uoa - Found entry's module UOA (UID or alias) cm_module_uid - Found entry's module UID cm_data_uoa - Found entry's UOA (UID or alias) cm_data_uid - Found entry's UID (cm_display_as_alias) - Fodun entry's user-friendly alias for printing } ... ] }
Search Input: { cm_es_string - simple search cm_es_query - elastic search query (cm_index_uoa) - index configuration (size) - limit return array } Output: { cm_return - return code >0 if error cm_array - array of UIDs }
def module::select | ( | i | ) |
Select from the choice Input: { cm_choices - TODO: describe format, see ct-compiler gcc-4.6.2 cm_exploration_strategy - random (choice_rnd_seed) - random seed (cm_choice) - previous cm_choice state (can be updated using various methods including probabilistic) } Output: { cm_return - return code = 0, if successful cm_choice }
def module::separate_common_from_data | ( | i | ) |
def module::server | ( | i | ) |
Index Server Input: { (cm_index_uoa) - uoa with description of indexer (os_uoa) - uoa with OS description (needed for script extension, for example) (cm_index_port) - web port } Output: { cm_return - return code >0 if error }
Simple Web Server Input: { cm_web_port - web port } Output: { cm_return - return code >0 if error }
def module::set_cpu_frequency | ( | i | ) |
Set CPU frequency Input: { (host_os_uoa) - host OS on which cM is running to detect which scripts to use for detection. If not set, the default one is taken from cM configuration. (target_os_uoa) - target OS to detect. If =='', host OS is used frequency - frequency (use_sudo) - if 'yes', use SUDO } Output: { cm_return - return code = 0, if successful system_state - [windows] [unix] }
def module::set_env | ( | i | ) |
def module::share | ( | i | ) |
def module::show | ( | i | ) |
Show file from the repository or tmp directory. Only web preprocess! Input: { cm_web_filename - file to show (cm_web_content) - extension known by cM to prepare html header. If not set, this function will try to detect it from the cm_web_filename extension (cm_module_uoa and cm_data_uoa) - entry to take file from. If not set, file is taken from the temporary directory (cm_repo_uoa) - repo UOA } Output: { cm_return - return code >0 if error cm_web_headers - web headers cm_web_postprocess=no - force cM not to postprocess }
def module::show_html | ( | i | ) |
Show file from the repository or tmp directory. Input: { cm_web_filename - file to show (cm_web_content) - extension known by cM to prepare html header. If not set, this function will try to detect it from the cm_web_filename extension (cm_module_uoa and cm_data_uoa) - entry to take file from. If not set, file is taken from the temporary directory } Output: { cm_return - return code >0 if error }
def module::show_legend | ( | i | ) |
def module::temporaly_allow_writing_in_kernel | ( | i | ) |
def module::test | ( | i | ) |
def module::test_repository | ( | i | ) |
Test repository (mainly remote access) It just try to load tmp:tmp and returns ok, if it exists or not found, thus meaning that the structure of repository/access is fine Input: { test_repo_uoa - repo UOA ... - various additional parameters (for example, for authentication) } Output: { cm_return - return code >0 if error }
def module::unregister_func | ( | i | ) |
def module::update_data | ( | i | ) |
Update data Input: { cm_run_module_uoa | cm_module_uoa - module UOA cm_data_uoa - data UOA (cm_data_uid) - force data UID (if UOA is alias) (cm_repo_uoa) - repo UOA (cm_console) - by default, do not output anything except errors if 'txt' then output as plain text if 'web' then output for web (html, image, etc) if 'json' then output as json cm_array - use ONLY this array to update data (cm_skip_update_info) - if yes, do not write 'cm_updated' (cm_update_only_index) - update only index (cm_file_upload) - text content of file for upload (cm_file_upload_base64) - file content as base64.urlsafe_b64encode (cm_file_upload_name) - preffered uploaded file name (cm_file_upload_type) - type of file (if .zip, it will be automatically unziped) (cm_archive) - 'none' 'already_archived_with_internal_name' - should save file under default cM name (cm_archive.zip) 'already_archived_with_user_name' - should save file under user name (cm_file_upload_tmp_uid) - normally generated automatically by web server when uploading files (cm_add_default) - if 'yes', add default params from descriptions (module/kernel) (cm_description) - description of the data entry (cm_display_as_alias) - user readable alias to display instead of module:data (cm_note) - note about this data entry (cm_like) - if 'yes' add +1 for this user to the entry ... - data to add to the repository if not set 'cm_array' (lock_uid) - UID of the aquired lock } Output: { cm_return - return code >0 if error -1 if entry exists cm_path - path to created data entry cm_uid - uid (from UOA) cm_alias - alias (from UOA) cm_uoa - alias or uid if alias=='' }
Update repository Input: { } Output: { }
Input: { } Output: { }
def module::update_default_kernel_data | ( | i | ) |
def module::update_input_from_state_ll | ( | ii, | ||
state, | ||||
p_uid, | ||||
execute = False | ||||
) |
def module::view_ctuning_space | ( | i | ) |
def module::view_data | ( | i | ) |
View data Input: { cm_module_uoa - module UOA cm_data_uoa - data UOA (cm_repo_uoa) - repo UOA (cm_console) - by default, do not output anything except errors if 'txt' then output as plain text if 'web' then output for web (html, image, etc) if 'json' then output as json (cm_back) - for back button } Output: { cm_return - return code >0 if error -1 if entry exists cm_path - path to data entry cm_uid - uid (from UOA) cm_alias - alias (from UOA) cm_data_obj - cM data obj (cm_data_obj['cfg'] - data entry json) }
def module::view_data_through_web | ( | i | ) |
View data through web Input: { browse_module_uoa - module UOA browse_data_uoa - data UOA (browse_repo_uoa) - repo UOA (cm_back_json) - for back button } Output: { cm_return - return code >0 if error -1 if entry exists cm_path - path to created data entry cm_uid - uid (from UOA) cm_alias - alias (from UOA) }
def module::view_stats | ( | i | ) |
def module::visualize | ( | i | ) |
def module::visualize_connections | ( | i | ) |
Detect parameters in web forms Input: { cm_show_only_graph - if 'yes', show only graph and skip all the rest cm_width - graph width cm_height - graph height } Output: { cm_return - return code >0 if error cm_array - parsed array with keys cm_actions - parsed array with commands }
def module::visualize_data | ( | i | ) |
Visualize data entry (with recursion) Input: { cm_array - array to visualize (cm_array_orig) - original array (not changing through the calls) (cm_data_desc) - data description for visualization (always first level) (cm_desc_state) - previous description in hierarchies (cm_form_commands) - form commands (cm_form_explore) - form explore parameters (cm_form_explore_color) - prefix for style cm_separator - appendable separator to deal with array hierarchy (full - can reproduce hierarchy) cm_separator_form - appendable separator to deal with array hierarchy (full and with additional forms to deal with commands) (cm_separator_desc) - appendable separator to deal with array hierarchy in descriptions (reduced - for example list number is skipped) (cm_mode) - 'add' or 'update' (cm_back_json) - data to create back button (show_param_names) - if =='on' show parameter names (cm_forms_exists) - if 'yes', processing through forms (to avoid reinit of default vars) (cm_support_raw_edit) - if 'yes', add check button to allow raw edit (cm_raw_edit) - if 'on', raw edit (cm_explore) - if 'yes', prepare exploration choices (cm_explore_commands) - array with exploration commands (cm_explore_state) - {} array with choices and exploration state (cm_explore_level) - exploration level ('' - coarse-grain, '1' - next level, ...) (hide_add_new_object) - if 'yes', do not show 'add new object' (ignore_desc_expand) - if 'yes', ignore data description expansion from other sources (otherwise can be very slow for viewing) } Output: { cm_return - return code >0 if error cm_string - html string cm_array - updated array to visualize cm_explore_state - updated explore state cm_data_desc - updated data description }
def module::visualize_selection | ( | i | ) |
Prepare graph data and call graph engine Input: { ct_module_uoa - module to visualize ct_data_selection - list of ctuning.space data ct_axes - axes ct_graph_type - graph engine ct_axes_desc - axes description (url) - starting url of cM (save_graph) - if 'yes', save graph (cm_file_ext) - file extension if save_graph=='yes' (reproduce_save_graph) - data to reproduce graph for saving (use_multiple_graphs) - if 'on' separate entries into different sub-graphs (convert_text_axes_to_float) - convert text axes to consequitive floats (mpl_extra_params) - if python mat plot is used, this dictionary has extra params mpl_image_size_x - image X size in inch mpl_image_size_y - image Y size in inch mpl_image_dpi - image DPI mpl_detach_graph - if 'on', detach graph (data_entry_to_sort) - if sort and multiple entries, use this entry number to sort (-1) (fix_additional_dimensions) - if 'yes', fix or range additional dimensions (fix_additional_dimensions_data) - dictionary of format: {"flattened key A":{'less':X, 'equal':Y, 'more':Z}, ...} } Output: { cm_return - return code >0 if error data to save graph, if save_graph=='yes' }
def module::web_build | ( | i | ) |
def module::web_configure | ( | i | ) |
def module::web_copy | ( | i | ) |
def module::web_detect | ( | i | ) |
def module::web_explore | ( | i | ) |
def module::web_install | ( | i | ) |
Install cM package/library When searching for already installed code, dependencies are checked * using classes for packages (to allow reuse of packages) * by exact code_uoa for code.source (to have exact build setup) FGG: the checks are a bit ugly and were done in a rush. Should be simplified, cleaned up and unified one day. Input: { package_data_uoa - UOA of package to install (package_repo_uoa) - repo of the package package_host_os_uoa - UOA of host OS (to set up script execution) (install_data_uid) - UID of the code where package will be installed if install_data_uid=='' it will be randomly generated (install_data_alias) - use this alias for a generated entry If package should be built: build_target_os_uoa - target OS uoa for building (code_deps) - list with code UOA for dependencies [{"index":"uoa"} ...] (run_target_processor_uoa) - target processor UOA (not strictly needed - can add some helper parameters before executing code) (run_set_env2) - array with environment variables to be set before executable (add_rem_to_script) - add rem to install script (skip_extract_and_build) - if 'yes', skip copy, extract and build (by setting CM_SKIP_BUILD=yes) } Output: { cm_return - return code = 0, if successful Output from 'code install' }
def module::web_run | ( | i | ) |
def module::web_view | ( | i | ) |
View aggregated experimental results Input: { TBD (web_title) - web title to substitute default (calculate_program_similarities) - if 'yes', calculate distance from programs based on properties (program_features) - use it to calcluate distance (pruning_similar_programs) - direct pruning parameters (number_of_returned_items) - number of returned items } Output: { cm_return - if =0, success (similar_programs) - list of similar programs (if not web) (similar_programs_points) - generated point per similar program (if not web) (similar_programs_points_improvements) - generated improvements for points per similar program (if not web) }
def module::web_view_data | ( | i | ) |
Prepare header Input: { cm_data - data object from pipeline header_desc - header description cur_line_style - { cur - cur (style1 or style2) style1 style2 } point - point to visualize } Output: { cm_return - if =0, success cm_html - header html }
Prepare header Input: { cm_data - data object from pipeline header_desc - header description cur_line_style - { cur - cur (style1 or style2) style1 style2 } point - point to visualize } Output: { cm_return - if =0, success cm_html - header html point - all values for the point point_improvements - all available improvements for the point }
def module::web_view_header | ( | i | ) |
def module::welcome | ( | i | ) |
string module::cm_action_index = 'index' |
string module::cm_action_show = 'show' |
string module::cm_data_uoa_default = 'default' |
module::cm_kernel = None |
string module::cm_menu_login = 'login' |
string module::cm_menu_logout = 'logout' |
tuple module::diter = int(space.get('number_of_explored_points','0')) |
string module::form6 = '#form6' |
module::ii = {} |
Definition at line 818 of file module.py.
Referenced by org::openme::openme::remote_access().
tuple module::ii_copy = copy.deepcopy(ii) |
dictionary module::ini = {} |
def module::init = state[pu['init']] |
list module::ip = state[pu['init_params']] |
dictionary module::jj |
tuple module::jj_copy = copy.deepcopy(jj) |
list module::lock_uid = '' |
list module::pd = state[pu['prepare_dataset']] |
Definition at line 781 of file module.py.
Referenced by org::openme::openme::access(), cm_action(), org::openme::openme::convert_cm_array_to_uri(), org::openme::openme::convert_str_to_sha1(), org::openme::openme::openme_load_json_file(), org::openme::openme::openme_store_json_file(), and org::openme::openme::remote_access().
module::recorded = True |
list module::rp = state[pu['run_program']] |
dictionary module::rr = {'cm_return':0} |
list module::su = cm_kernel.ini['dcfg'] |
string module::view_and_sort = 'cm_display_html' |