English | 中文
How to add a new model on FastDeploy, including C++/Python deployment? Here, we take the ResNet50 model in torchvision v0.12.0 as an example, introducing external Model Integration on FastDeploy. The whole process only needs 3 steps.
Step | Description | Create or modify the files |
---|---|---|
1 | Add a model implementation to the corresponding task module in FastDeploy/vision | resnet.h, resnet.cc, vision.h |
2 | Python interface binding via pybind | resnet_pybind.cc, classification_pybind.cc |
3 | Use Python to call Interface | resnet.py, __init__.py |
After completing the above 3 steps, an external model is integrated.
If you want to contribute your code to FastDeploy, it is very kind of you to add test code, instructions (Readme), and code annotations for the added model in the test.
Before integrating external models, it is important to convert the trained models (.pt, .pdparams, etc.) to the model formats (.onnx, .pdmodel) that FastDeploy supports for deployment. Most open source repositories provide model conversion scripts for developers. As torchvision does not provide conversion scripts, developers can write conversion scripts manually. In this demo, we convert torchvison.models.resnet50
to resnet50.onnx
with the following code for your reference.
import torch
import torchvision.models as models
model = models.resnet50(pretrained=True)
batch_size = 1 #batch size
input_shape = (3, 224, 224) #Input data, change to your own input shape
model.eval()
x = torch.randn(batch_size, *input_shape) # Generate Tensor
export_onnx_file = "resnet50.onnx" # ONNX file name
torch.onnx.export(model,
x,
export_onnx_file,
opset_version=12,
input_names=["input"], # Input names
output_names=["output"], # Output names
dynamic_axes={"input":{0:"batch_size"}, # batch size variables
"output":{0:"batch_size"}})
Running the above script will generate aresnet50.onnx
file.
resnet.h
file
Predict
, Initialize
, Preprocess
, Postprocess
and Constructor
, and necessary variables, please refer to resnet.h for details.class FASTDEPLOY_DECL ResNet : public FastDeployModel {
public:
ResNet(...);
virtual bool Predict(...);
private:
bool Initialize();
bool Preprocess(...);
bool Postprocess(...);
};
resnet.cc
file
resnet.h
to resnet.cc
, where PreProcess
and PostProcess
need to refer to the official source library for pre- and post-processing logic reproduction. The specific logic of each ResNet function is as follows. For more detailed code, please refer to resnet.cc.ResNet::ResNet(...) {
// Constructor logic
// 1. Specify Backend 2. Set RuntimeOption 3. Call Initialize()function
}
bool ResNet::Initialize() {
// Initialization logic
// 1. Assign values to global variables 2. Call InitRuntime()function
return true;
}
bool ResNet::Preprocess(Mat* mat, FDTensor* output) {
// Preprocess logic
// 1. Resize 2. BGR2RGB 3. Normalize 4. HWC2CHW 5. save the results to FDTensor class
return true;
}
bool ResNet::Postprocess(FDTensor& infer_result, ClassifyResult* result, int topk) {
//Postprocess logic
// 1. Softmax 2. Choose topk labels 3. Save the results to ClassifyResult
return true;
}
bool ResNet::Predict(cv::Mat* im, ClassifyResult* result, int topk) {
Preprocess(...)
Infer(...)
Postprocess(...)
return true;
}
vision.h
#ifdef ENABLE_VISION
#include "fastdeploy/vision/classification/contrib/resnet.h"
#endif
Create Pybind file
Create path
Create content
Use Pybind to bind function variables from C++ to Python, please refer to resnet_pybind.cc for more details.
void BindResNet(pybind11::module& m) {
pybind11::class_<vision::classification::ResNet, FastDeployModel>(
m, "ResNet")
.def(pybind11::init<std::string, std::string, RuntimeOption, ModelFormat>())
.def("predict", ...)
.def_readwrite("size", &vision::classification::ResNet::size)
.def_readwrite("mean_vals", &vision::classification::ResNet::mean_vals)
.def_readwrite("std_vals", &vision::classification::ResNet::std_vals);
}
Call Pybind function
modify path
modify content
void BindResNet(pybind11::module& m);
void BindClassification(pybind11::module& m) {
auto classification_module =
m.def_submodule("classification", "Image classification models.");
BindResNet(classification_module);
}
resnet.py
file
\_\_init\_\_
, Pybind bonded functions (such as predict()
), and functions to assign and get global variables bound to Pybind
. Please refer to resnet.py for detailsclass ResNet(FastDeployModel):
def __init__(self, ...):
self._model = C.vision.classification.ResNet(...)
def predict(self, input_image, topk=1):
return self._model.predict(input_image, topk)
@property
def size(self):
return self._model.size
@size.setter
def size(self, wh):
...
from .contrib.resnet import ResNet
mkdir build & cd build
cmake .. -DENABLE_ORT_BACKEND=ON -DENABLE_VISION=ON -DCMAKE_INSTALL_PREFIX=${PWD/fastdeploy-0.0.3
-DENABLE_PADDLE_BACKEND=ON -DENABLE_TRT_BACKEND=ON -DWITH_GPU=ON -DTRT_DIRECTORY=/PATH/TO/TensorRT/
make -j8
make install
Compile to get build/fastdeploy-0.0.3/。
export TRT_DIRECTORY=/PATH/TO/TensorRT/ #If TensorRT is used, developers need to fill in the location of TensorRT and enable ENABLE_TRT_BACKEND
export ENABLE_TRT_BACKEND=ON
export WITH_GPU=ON
export ENABLE_PADDLE_BACKEND=ON
export ENABLE_OPENVINO_BACKEND=ON
export ENABLE_VISION=ON
export ENABLE_ORT_BACKEND=ON
python setup.py build
python setup.py bdist_wheel
cd dist
pip install fastdeploy_gpu_python-Version number-cpxx-cpxxm-system architecture.whl
.
├── cpp
│ ├── CMakeLists.txt
│ ├── infer.cc // C++ test code
│ └── README.md // C++ Readme
├── python
│ ├── infer.py // Python test code
│ └── README.md // Python Readme
└── README.md // ResNet model integration readme
mkdir build & cd build
cmake .. -DFASTDEPLOY_INSTALL_DIR=/PATH/TO/FastDeploy/build/fastdeploy-0.0.3/
make
To make the code clear for understanding, developers can annotate the newly-added code.
/** \brief Predict for the input "im", the result will be saved in "result".
*
* \param[in] im Input image for inference.
* \param[in] result Saving the inference result.
* \param[in] topk The length of return values, e.g., if topk==2, the result will include the 2 most possible class label for input image.
*/
virtual bool Predict(cv::Mat* im, ClassifyResult* result, int topk = 1);
/// Tuple of (width, height)
std::vector<int> size;
/*! @brief Initialize for ResNet model, assign values to the global variables and call InitRuntime()
*/
bool Initialize();
def predict(self, input_image, topk=1):
"""Classify an input image
:param input_image: (numpy.ndarray)The input image data, 3-D array with layout HWC, BGR format
:param topk: (int)The topk result by the classify confidence score, default 1
:return: ClassifyResult
"""
return self._model.predict(input_image, topk)
Other files in the integration process can also be annotated to explain the details of the implementation.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。