Docs
Search…
Python Tensorflow Example (GPU)
In this Tutorial you'll learn how to use GPU calculation in a Python module in order to improve the performance of your module.

Introduction

This tutorial only works with the Linux OS as it is the only one supporting the nvidia-docker plugin.

Why using GPU?

Machine Learning (ML) applications like classification, segmentation, object detection, etc. usually have a high demand of computing power. GPUs most of the time help to to speed up machine learning e.g. the training procedure on big data sets.
APEER platform provides you with an easy way to use GPU power remotely with only minor changes in comparison to the basic CPU usage.

How to move from CPU to GPU?

In order to reinforce your modules with GPU resources you need to modify three files that come with all APEER modules:
  • Dockerfile Change the first line of the code, that describes the docker base image, i.e. a docker environment that already contains some libraries and frameworks:
    1
    FROM tensorflow/tensorflow:1.12.0-gpu-py3
    Copied!
  • requirements.txt Add libraries which are needed to the module's task. For example in order to use tensorflow-gpu library together with Keras, we need to add the following libraries:
    1
    tensorflow-gpu==1.12.0
    2
    Keras==2.2.2
    3
    Keras-Applications==1.0.4
    4
    Keras-Preprocessing==1.0.2
    Copied!
  • module_specification.json There is a need to add the 'requirements' key nested in the first layer of the json file, together with 'spec' and 'ui'.
    1
    "requirements": {
    2
    "hardware": {
    3
    "gpu": {}
    4
    }
    5
    }
    Copied!
After the mentioned changes to your module will already be able to run on GPUs provided on APEER platform.
Here you find a list of useful links explaining the libraries commonly used for solving Machine Learning tasks:
  • TensorFlow: open source library with a broad functionality for Machine Learning.
  • Keras: high-level API for building, training and testing Neural Networks. Keras allows to define your ML models in a quick and easy way.
  • Theano: Python library with very efficient functionality for working with big high-dimensional data. In terms of control over the code can be compared to TensorFlow but might take more time to get first results in comparison to Keras.
  • TensorLayer: extended version of TensorFlow. It provides with popular Deep Learning models, a broad range of loss functions and metrics. Can be easily integrated together with TensorFlow and Keras.
  • Caffe: yet another Deep Learning library.
Let's jump into practice and have a look on the Supervised Segmentation Trainer module as an example of the Python GPU module. The module requires TensorFlow, Keras, TensorLayer libraries.

Python Tensorflow-GPU Example Module

Writing the module

You can download the source code from our GitHub repository or by creating a new 'Python GPU Example' Module in APEER.
The goal of the Supervised Segmentation Trainer module is to train the classification model based on the set of input raw images, the corresponding ground truth labels and several hyperparameters (e.g. learning rate). The module will generate a trained model (model.h5) and a history of the learning curve (history.json).
As learned in the introduction you need to change the Dockerfile, requiremtents.txt and module_specification.json. The prepared files tuned for this specific GPU example are listed below:
Dockerfile
requirements.txt
module_specification.json
1
FROM tensorflow/tensorflow:1.12.0-gpu-py3
2
3
WORKDIR /usr/src/app
4
5
COPY requirements.txt .
6
RUN pip install --no-cache-dir -r requirements.txt
7
8
COPY apeer_main.py .
9
COPY train_classification.py .
10
COPY classification_model.py .
11
COPY module_specification.json .
12
13
ENTRYPOINT [ "python", "./apeer_main.py" ]
Copied!
1
apeer-dev-kit>=1.0.6,<2
2
apeer-ometiff-library>=1.3.1,<2
3
4
scipy==1.1.0
5
numpy==1.15.4
6
7
tensorlayer==1.11.1
8
tensorflow-gpu==1.12.0
9
10
Keras==2.2.2
11
Keras-Applications==1.0.4
12
Keras-Preprocessing==1.0.2
13
Copied!
1
{
2
"spec": {
3
"inputs": {
4
"images_files": {
5
"type:list[file]": {}
6
},
7
"labels_file": {
8
"type:file": {}
9
},
10
"learn_rate": {
11
"type:number":{},
12
"default": 1e-4
13
},
14
"batch_size": {
15
"type:integer": {
16
"min": 1,
17
"max": 512
18
},
19
"default": 4
20
},
21
"num_epochs": {
22
"type:integer": {
23
"min": 1,
24
"max": 500
25
},
26
"default": 20
27
},
28
"valid_split_ratio": {
29
"type:number": {},
30
"default": 0.1
31
},
32
"pretrained_model": {
33
"type:string": {},
34
"default": ""
35
}
36
},
37
"outputs": {
38
"trained_model": {
39
"type:file": {}
40
},
41
"train_history": {
42
"type:file": {}
43
}
44
}
45
},
46
"ui": {
47
"inputs": {
48
"images_files": {
49
"index": 0,
50
"label": "Input images",
51
"description": "The list of input raw images",
52
"widget:none": null
53
},
54
"labels_file": {
55
"index": 1,
56
"label": "Input labels",
57
"description": "The list of input ground truth labels",
58
"widget:none": null
59
},
60
"learn_rate": {
61
"index": 2,
62
"label": "Learning rate",
63
"description": "The learning rate for ADAM optimizer",
64
"widget:textbox": {}
65
},
66
"batch_size": {
67
"index": 3,
68
"label": "Batch size",
69
"description": "The amount of images to be processed in one iteration of training",
70
"widget:textbox": {}
71
},
72
"num_epochs": {
73
"index": 4,
74
"label": "Epochs",
75
"description": "The number of epochs for model to train",
76
"widget:textbox": {}
77
},
78
"valid_split_ratio": {
79
"index": 5,
80
"label": "Validation split ratio",
81
"description": "Fraction of the training data to be used as validation data",
82
"widget:textbox": {}
83
},
84
"pretrained_model": {
85
"index": 6,
86
"label": "Pretrained model",
87
"description": "Pretrained segmentation model",
88
"widget:none": null
89
}
90
},
91
"outputs": {
92
"trained_model": {
93
"index": 7,
94
"label": "Trained model",
95
"description": "Trained model together with its structure and weights"
96
},
97
"train_history": {
98
"index": 8,
99
"label": "Train history",
100
"description": "JSON file with an information about training progress through epochs"
101
}
102
}
103
},
104
"requirements": {
105
"hardware": {
106
"gpu": {}
107
}
108
}
109
}
Copied!
Let us have a closer look on the other module files. In general we have to conduct four steps in order to make use of a machine learning model:
  1. 1.
    Read the input data
  2. 2.
    Normalize data
  3. 3.
    Loading and training the model
  4. 4.
    Saving the results
train_classification.py is not only the main file of the module it also takes care about the above mentioned four steps:
train_classification.py
1
from __future__ import absolute_import
2
from __future__ import division
3
from __future__ import print_function
4
5
import numpy as np
6
import skimage.io
7
import os
8
import json
9
from keras.utils import to_categorical
10
from keras.optimizers import Adam
11
from keras.models import load_model
12
13
from classification_model import get_model
14
15
16
def execute(images_files, labels_file, learn_rate=1e-4, batch_size=16, num_epochs=100, valid_split_ratio=0.1,
17
pretrained_model='', out_model_file='/output/model.h5', out_history_file='/output/history.json'):
18
"""Trains fully supervised classification model and saves the trained model
19
Args:
20
images_files: list of strings. paths to the input raw images
21
labels_file: string. path to the csv file with corresponding labels for the raw images
22
learn_rate: float. learning rate
23
batch_size: int. the amount of images to be processed in one iteration of training
24
num_epochs: int. the number of epochs for model to train
25
valid_split_ratio: float. fraction of the training data to be used as validation data
26
pretrained_model: string. the path to the pretrained segmentation model
27
out_model_file: string. the path where the trained model will be saved
28
Returns:
29
dict. key: 'trained_model', value: string - path to the trained model
30
key: 'train_history', value: string - path to the training history
31
"""
32
33
# Store input images into numpy array
34
images = []
35
for image_file in images_files:
36
if image_file.endswith('json'):
37
continue
38
images.append(skimage.io.imread(image_file))
39
images = np.asarray(images, dtype='float32')
40
41
# In case input images are grayscale -
42
# add 1 more dimension to specify that the channel amount is 1
43
if images.ndim == 3:
44
images = np.expand_dims(images, axis=-1)
45
46
# Standard normalization of raw images per channel
47
mean = np.mean(images, axis=(0, 1, 2), keepdims=True)
48
std = np.std(images, axis=(0, 1, 2), keepdims=True)
49
images -= mean
50
images /= std
51
52
labels = np.genfromtxt(labels_file)
53
54
# Sets the amount of different classes in the ground truth
55
num_classes = int(labels.max()) + 1
56
57
# Convert ground truth masks to one hot encoding format.
58
masks_one_hot = to_categorical(labels, num_classes)
59
60
channels_num = images.shape[-1]
61
img_shape = (images.shape[1], images.shape[2], channels_num)
62
63
# If pretrained model is given - load it, otherwise - build randomly initialized UNet
64
if pretrained_model != '':
65
assert os.path.isfile(pretrained_model), 'Wrong pretrained model file'
66
model = load_model(pretrained_model)
67
else:
68
model = get_model(img_shape, num_classes=num_classes)
69
70
# List of metrics to be used as an evaluation of the model during the train time
71
metrics = ['accuracy']
72
73
# Compile the model specifying the optimizer and the loss function
74
model.compile(optimizer=Adam(lr=learn_rate), loss='categorical_crossentropy', metrics=metrics)
75
76
# Train the model
77
progress = model.fit(images, masks_one_hot, batch_size=batch_size, epochs=num_epochs,
78
verbose=True, validation_split=valid_split_ratio, shuffle=True)
79
80
# create '/output' folder if it does not exist already
81
if not os.path.isdir('output'):
82
os.mkdir('output')
83
84
# Save the model together with its structure and trained weights
85
model.save(out_model_file)
86
87
# Save the training history as a json file
88
with open(out_history_file, 'w') as file:
89
json.dump(progress.history, file)
90
91
return {'trained_model': out_model_file,
92
'train_history': out_history_file}
93
94
95
# Test code locally
96
if __name__ == "__main__":
97
images_files = ['input/img_0_0.jpg',
98
'input/img_0_1.jpg',
99
'input/img_0_2.jpg',
100
'input/img_0_3.jpg',
101
'input/img_0_4.jpg',
102
'input/img_1_0.jpg',
103
'input/img_1_1.jpg',
104
'input/img_1_2.jpg',
105
'input/img_1_3.jpg',
106
'input/img_1_4.jpg']
107
labels_file = 'input/labels.csv'
108
109
learn_rate = 1e-4
110
batch_size = 1
111
num_epochs = 4
112
valid_split_ratio = 0.1
113
pretrained_model = ''
114
out_model_file = 'output/trained_model.h5'
115
out_history_file = 'output/history.json'
116
117
execute(images_files, labels_file, learn_rate, batch_size, num_epochs, valid_split_ratio,
118
pretrained_model, out_model_file, out_history_file)
Copied!
In line 13 of the previous script (from classification_model import get_model) you can notice that we import the structure of the model from classification_model.py. This code is responsible for building the structure of a very simplified keras classification model. Most of the computer vision problems are solved by using convolutional neuronal networks. In our example case we will use only 2 convolutional layers followed by 1 dense layer with the softmax activation.
classification_model.py
1
from __future__ import absolute_import
2
from __future__ import division
3
from __future__ import print_function
4
5
from keras.models import Model
6
from keras.layers import Input, Conv2D, Flatten, Dense
7
from keras import backend as K
8
9
K.set_image_data_format('channels_last')
10
11
def get_model(img_shape, num_classes=2):
12
inputs = Input(shape=img_shape)
13
14
conv1_1 = Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(inputs)
15
conv1_2 = Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv1_1)
16
flat1 = Flatten()(conv1_2)
17
dense1 = Dense(num_classes, activation='softmax')(flat1)
18
19
model = Model(inputs=inputs, output=[dense1])
20
21
return model
Copied!
apeer_main.py also has to be modified with the respect to the change of input and output parameters.
apeer_main.py
1
from apeer_dev_kit import adk
2
import train_classification
3
4
5
if __name__ == "__main__":
6
inputs = adk.get_inputs()
7
8
outputs = train_classification.execute(inputs['images_files'], inputs['labels_file'],
9
inputs['learn_rate'], inputs['batch_size'],
10
inputs['num_epochs'], inputs['valid_split_ratio'],
11
inputs['pretrained_model'])
12
13
adk.set_file_output('trained_model', outputs['trained_model'])
14
adk.set_file_output('train_history', outputs['train_history'])
15
adk.finalize()
Copied!

Testing the module

To test this particular module locally you have two ways to do it:
  1. 1.
    Running train_classification.py directly on the CPU (no changes required)
  2. 2.
    Using nvidia-docker container with enabled GPU usage
The former way needs some preparation. Thus, docker_make.sh file is modified to activate nvidia-docker instead of simple docker:
docker_make.sh
1
#!/bin/bash
2
3
nvidia-docker build -t 'apeer/train_classification' .
4
5
nvidia-docker run -it --rm -v "$(pwd)"/input:/input -v "$(pwd)"/output:/output -e "WFE_INPUT_JSON=$(<wfe.env)" apeer/train_classification
Copied!
Furthermore, wfe.env has to be changed too, as it contains the input parameters we normally fill out on APEER before running the workflow:
wfe.env
1
{
2
"WFE_output_params_file":"wfe_module_params_1_1.json",
3
"images_files":["input/img_0_0.jpg", "input/img_0_1.jpg", "input/img_0_2.jpg", "input/img_0_3.jpg", "input/img_0_4.jpg", "input/img_1_0.jpg", "input/img_1_1.jpg", "input/img_1_2.jpg", "input/img_1_3.jpg", "input/img_1_4.jpg"],
4
"labels_file":"input/labels.csv",
5
"learn_rate":1e-4,
6
"batch_size":1,
7
"num_epochs":4,
8
"valid_split_ratio":0.1,
9
"pretrained_model": ""
10
}
Copied!

Fast lane for impatient coders ;)

Of course we have prepared the zipped project folder and some ready to run files for copy and paste:
apeer_main.py
classification_model.py
docker_make.sh
Dockerfile
module_specification.json
requirements.txt
train_classification.py
wfe.env
1
from apeer_dev_kit import adk
2
import train_classification
3
4
5
if __name__ == "__main__":
6
inputs = adk.get_inputs()
7
8
outputs = train_classification.execute(inputs['images_files'], inputs['labels_file'],
9
inputs['learn_rate'], inputs['batch_size'],
10
inputs['num_epochs'], inputs['valid_split_ratio'],
11
inputs['pretrained_model'])
12
13
adk.set_file_output('trained_model', outputs['trained_model'])
14
adk.set_file_output('train_history', outputs['train_history'])
15
adk.finalize()
16
Copied!
1
from __future__ import absolute_import
2
from __future__ import division
3
from __future__ import print_function
4
5
from keras.models import Model
6
from keras.layers import Input, Conv2D, Flatten, Dense
7
from keras import backend as K
8
9
K.set_image_data_format('channels_last')
10
11
def get_model(img_shape, num_classes=2):
12
inputs = Input(shape=img_shape)
13
14
conv1_1 = Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(inputs)
15
conv1_2 = Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv1_1)
16
flat1 = Flatten()(conv1_2)
17
dense1 = Dense(num_classes, activation='softmax')(flat1)
18
19
model = Model(inputs=inputs, output=[dense1])
20
21
return model
Copied!
1
#!/bin/bash
2
3
nvidia-docker build -t 'apeer/train_classification' .
4
5
nvidia-docker run -it --rm -v "$(pwd)"/input:/input -v "$(pwd)"/output:/output -e "WFE_INPUT_JSON=$(<wfe.env)" apeer/train_classification
6
Copied!
1
FROM tensorflow/tensorflow:1.12.0-gpu-py3
2
3
4
WORKDIR /usr/src/app
5
6
COPY requirements.txt .
7
RUN pip install --no-cache-dir -r requirements.txt
8
9
COPY apeer_main.py .
10
COPY train_classification.py .
11
COPY classification_model.py .
12
COPY module_specification.json .
13
14
ENTRYPOINT [ "python", "./apeer_main.py" ]
Copied!
1
{
2
"spec": {
3
"inputs": {
4
"images_files": {
5
"type:list[file]": {}
6
},
7
"labels_file": {
8
"type:file": {}
9
},
10
"learn_rate": {
11
"type:number":{},
12
"default": 1e-4
13
},
14
"batch_size": {
15
"type:integer": {
16
"min": 1,
17
"max": 512
18
},
19
"default": 4
20
},
21
"num_epochs": {
22
"type:integer": {
23
"min": 1,
24
"max": 500
25
},
26
"default": 20
27
},
28
"valid_split_ratio": {
29
"type:number": {},
30
"default": 0.1
31
},
32
"pretrained_model": {
33
"type:string": {},
34
"default": ""
35
}
36
},
37
"outputs": {
38
"trained_model": {
39
"type:file": {}
40
},
41
"train_history": {
42
"type:file": {}
43
}
44
}
45
},
46
"ui": {
47
"inputs": {
48
"images_files": {
49
"index": 0,
50
"label": "Input images",
51
"description": "The list of input raw images",
52
"widget:none": null
53
},
54
"labels_file": {
55
"index": 1,
56
"label": "Input labels",
57
"description": "The list of input ground truth labels",
58
"widget:none": null
59
},
60
"learn_rate": {
61
"index": 2,
62
"label": "Learning rate",
63
"description": "The learning rate for ADAM optimizer",
64
"widget:textbox": {}
65
},
66
"batch_size": {
67
"index": 3,
68
"label": "Batch size",
69
"description": "The amount of images to be processed in one iteration of training",
70
"widget:textbox": {}
71
},
72
"num_epochs": {
73
"index": 4,
74
"label": "Epochs",
75
"description": "The number of epochs for model to train",
76
"widget:textbox": {}
77
},
78
"valid_split_ratio": {
79
"index": 5,
80
"label": "Validation split ratio",
81
"description": "Fraction of the training data to be used as validation data",
82
"widget:textbox": {}
83
},
84
"pretrained_model": {
85
"index": 6,
86
"label": "Pretrained model",
87
"description": "Pretrained segmentation model",
88
"widget:none": null
89
}
90
},
91
"outputs": {
92
"trained_model": {
93
"index": 7,
94
"label": "Trained model",
95
"description": "Trained model together with its structure and weights"
96
},
97
"train_history": {
98
"index": 8,
99
"label": "Train history",
100
"description": "JSON file with an information about training progress through epochs"
101
}
102
}
103
},
104
"requirements": {
105
"hardware": {
106
"gpu": {}
107
}
108
}
109
}
110
Copied!
1
apeer-dev-kit>=1.0.6,<2
2
apeer-ometiff-library>=1.3.1,<2
3
4
scipy==1.1.0
5
numpy==1.15.4
6
7
tensorlayer==1.11.1
8
tensorflow-gpu==1.12.0
9
10
Keras==2.2.2
11
Keras-Applications==1.0.4
12
Keras-Preprocessing==1.0.2
13
Copied!
1
from __future__ import absolute_import
2
from __future__ import division
3
from __future__ import print_function
4
5
import numpy as np
6
import skimage.io
7
import os
8
import json
9
from keras.utils import to_categorical
10
from keras.optimizers import Adam
11
from keras.models import load_model
12
13
from classification_model import get_model
14
15
16
def execute(images_files, labels_file, learn_rate=1e-4, batch_size=16, num_epochs=100, valid_split_ratio=0.1,
17
pretrained_model='', out_model_file='/output/model.h5', out_history_file='/output/history.json'):
18
"""Trains fully supervised classification model and saves the trained model
19
Args:
20
images_files: list of strings. paths to the input raw images
21
labels_file: string. path to the csv file with corresponding labels for the raw images
22
learn_rate: float. learning rate
23
batch_size: int. the amount of images to be processed in one iteration of training
24
num_epochs: int. the number of epochs for model to train
25
valid_split_ratio: float. fraction of the training data to be used as validation data
26
pretrained_model: string. the path to the pretrained segmentation model
27
out_model_file: string. the path where the trained model will be saved
28
Returns:
29
dict. key: 'trained_model', value: string - path to the trained model
30
key: 'train_history', value: string - path to the training history
31
"""
32
33
# Store input images into numpy array
34
images = []
35
for image_file in images_files:
36
if image_file.endswith('json'):
37
continue
38
images.append(skimage.io.imread(image_file))
39
images = np.asarray(images, dtype='float32')
40
41
# In case input images are grayscale -
42
# add 1 more dimension to specify that the channel amount is 1
43
if images.ndim == 3:
44
images = np.expand_dims(images, axis=-1)
45
46
# Standard normalization of raw images per channel
47
mean = np.mean(images, axis=(0, 1, 2), keepdims=True)
48
std = np.std(images, axis=(0, 1, 2), keepdims=True)
49
images -= mean
50
images /= std
51
52
labels = np.genfromtxt(labels_file)
53
54
# Sets the amount of different classes in the ground truth
55
num_classes = int(labels.max()) + 1
56
57
# Convert ground truth masks to one hot encoding format.
58
masks_one_hot = to_categorical(labels, num_classes)
59
60
channels_num = images.shape[-1]
61
img_shape = (images.shape[1], images.shape[2], channels_num)
62
63
# If pretrained model is given - load it, otherwise - build randomly initialized UNet
64
if pretrained_model != '':
65
assert os.path.isfile(pretrained_model), 'Wrong pretrained model file'
66
model = load_model(pretrained_model)
67
else:
68
model = get_model(img_shape, num_classes=num_classes)
69
70
# List of metrics to be used as an evaluation of the model during the train time
71
metrics = ['accuracy']
72
73
# Compile the model specifying the optimizer and the loss function
74
model.compile(optimizer=Adam(lr=learn_rate), loss='categorical_crossentropy', metrics=metrics)
75
76
# Train the model
77
progress = model.fit(images, masks_one_hot, batch_size=batch_size, epochs=num_epochs,
78
verbose=True, validation_split=valid_split_ratio, shuffle=True)
79
80
# create '/output' folder if it does not exist already
81
if not os.path.isdir('output'):
82
os.mkdir('output')
83
84
# Save the model together with its structure and trained weights
85
model.save(out_model_file)
86
87
# Save the training history as a json file
88
with open(out_history_file, 'w') as file:
89
json.dump(progress.history, file)
90
91
return {'trained_model': out_model_file,
92
'train_history': out_history_file}
93
94
95
# Test code locally
96
if __name__ == "__main__":
97
images_files = ['input/img_0_0.jpg',
98
'input/img_0_1.jpg',
99
'input/img_0_2.jpg',
100
'input/img_0_3.jpg',
101
'input/img_0_4.jpg',
102
'input/img_1_0.jpg',
103
'input/img_1_1.jpg',
104
'input/img_1_2.jpg',
105
'input/img_1_3.jpg',
106
'input/img_1_4.jpg']
107
labels_file = 'input/labels.csv'
108
109
learn_rate = 1e-4
110
batch_size = 1
111
num_epochs = 4
112
valid_split_ratio = 0.1
113
pretrained_model = ''
114
out_model_file = 'output/trained_model.h5'
115
out_history_file = 'output/history.json'
116
117
execute(images_files, labels_file, learn_rate, batch_size, num_epochs, valid_split_ratio,
118
pretrained_model, out_model_file, out_history_file)
119
Copied!
1
{
2
"WFE_output_params_file":"wfe_module_params_1_1.json",
3
"images_files":["/input/img_0_0.jpg", "/input/img_0_1.jpg", "/input/img_0_2.jpg", "/input/img_0_3.jpg", "/input/img_0_4.jpg", "/input/img_1_0.jpg", "/input/img_1_1.jpg", "/input/img_1_2.jpg", "/input/img_1_3.jpg", "/input/img_1_4.jpg"],
4
"labels_file":"/input/labels.csv",
5
"learn_rate":1e-4,
6
"batch_size":1,
7
"num_epochs":4,
8
"valid_split_ratio":0.1,
9
"pretrained_model": ""
10
}
Copied!
You can also download the source code by creating a new 'Python GPU Example' Module in APEER.
Python GPU Example.zip
1MB
Binary
Python Tensorflow-GPU Example Source Code

You couldn't find all the information you need? Contact us!

If you need help, just check out our FAQ. For any further questions please contact us at [email protected] or have a look at How-tos section in our blog or follow us on Twitter to stay up to date.
Last modified 2yr ago