作者:润和软件 郭新星
相机作为智能手机上少有的成长空间不错的,能够做出差异化的功能,每年都能成为各大android手机厂商争相宣传的亮点。众所周知android采用linux 作为其内核,而linux采用的开源协议具有传染性[1],导致android hal[2]成为了手机厂商们竞争的重要战场。随着openharmony 3.1[3]的发布,相机模块也逐渐完善起来,目前提供了基础预览和拍照的能力。openharmony中,相机用户态驱动框架承担了和android camera hal一样的角色,这部分位于openharmony的hdf[4]中,对上实现相机hdi[5]接口,对下实现相机pipeline模型,管理相机各个硬件设备。
相机用户态驱动框架(下图的camerahost 部分)总体可以分为三层,hdi实现层,实现相机标准南向接口;框架层,对接hdi实现层的控制、流的转发,实现数据通路的搭建、管理相机各个硬件设备等功能;适配层,屏蔽底层芯片和os差异,支持多平台适配。
模块介绍
hdi implementation:对上实现hdi接口,向下调用框架层的接口,完成hdi接口任务的转发。
buffer manager :屏蔽不同内存管理的差异,为子系统提供统一的操作接口,同时提供buffer轮转的功能。
pipeline core :解析hcs配置完成pipeline的搭建,调度pipeline中的各个node完成流的处理
device manager:通过调用底层硬件适配层接口,实现查询控制底层设备、枚举监听底层设备的功能。
platform adaption :屏蔽硬件差异,为device manager提供统一的操作底层硬件的能力。
目录结构
shelldrivers/peripheral/camera|-- readme_zh.md|-- bundle.json|-- figures| `-- logic-view-of-modules-related-to-this-repository_zh.png|-- hal| |-- build.gn| |-- adapter| |-- buffer_manager| |-- camera.gni| |-- device_manager| |-- hdi_impl| |-- include| |-- init| |-- pipeline_core| |-- test| `-- utils|-- hal_c| |-- build.gn| |-- camera.gni| |-- hdi_cif| `-- include`-- interfaces |-- hdi_ipc |-- hdi_passthrough `-- include (左右移动查看全部内容)
hdi implementation中的预览流程
接下来我们通过已经发布的openharmony 3.1开源代码,来看看预览是怎么完成的吧
drivers/peripheral/camera/hal/test/v4l2/src /preview_test.cpp存放了针对v4l2的预览测试代码,入口如下:
c++test_f(utestpreviewtest, camera_preview_0001){ std::cout << ==========[test log] preview stream, expected success.
intents = {camera::preview}; // 预览流 display_->startstream(display_->intents); // 起流 // get preview display_->startcapture(display_->streamid_preview, display_->captureid_preview, false, true); // release stream display_->captureids = {display_->captureid_preview}; display_->streamids = {display_->streamid_preview}; display_->stopstream(display_->captureids, display_->streamids);} (左右移动查看全部内容)
先获取stream operator实例
c++void testdisplay::achievestreamoperator(){ // create and get streamoperator information std::shared_ptr streamoperatorcallback = std::make_shared(); rc = cameradevice->getstreamoperator(streamoperatorcallback, streamoperator); // ........} (左右移动查看全部内容)
通过前文的streamoperator创建流
c++void testdisplay::startstream(std::vector intents){ // .............................. for (auto& intent : intents) { if (intent == 0) { std::shared_ptr producer = ibufferproducer::createbufferqueue(); producer->setqueuesize(8); // 创建buffer的生产端,并和相应的流进行绑定 auto callback = [this](std::shared_ptr prebuffer) { buffercallback(prebuffer, preview_mode); return; }; producer->setcallback(callback); streaminfo->streamid_ = streamid_preview; streaminfo->width_ = 640; // 640:picture width streaminfo->height_ = 480; // 480:picture height streaminfo->format_ = camera_format_yuyv_422_pkg; streaminfo->datasapce_ = 8; // 8:picture datasapce streaminfo->intent_ = intent; streaminfo->tunneledmode_ = 5; // 5:tunnel mode streaminfo->bufferqueue_ = producer; streaminfos.push_back(streaminfo); } else if (intent == 1) { // ....................... } rc = streamoperator->createstreams(streaminfos); // 创建流 // ................................ rc = streamoperator->commitstreams(camera::normal, ability); // 提交流 // .................................} (左右移动查看全部内容)
下面我们正式进入到hal的源代码中看看是怎么创建流的吧
c++camretcode streamoperator::createstreams(const std::vector& streaminfos){ // ..... for (auto it : streaminfos) {//.... std::shared_ptr stream = streamfactory::instance().createshared( istream::g_availablestreamtype[it->intent_], it->streamid_, it->intent_, pipelinecore_, messenger_); // 创建流实例// ... streamconfiguration scg; scg.id = it->streamid_; scg.type = it->intent_; scg.width = it->width_; scg.height = it->height_; pixelformat pf = static_cast(it->format_); scg.format = bufferadapter::pixelformattocameraformat(pf); scg.dataspace = it->datasapce_; scg.tunnelmode = it->tunneledmode_; scg.minframeduration = it->minframeduration_; scg.encodetype = it->encodetype_; retcode rc = stream->configstream(scg); // 依据上文的流信息配置流// ... if (it->bufferqueue_ != nullptr) { // 绑定前文的生产端 auto tunnel = std::make_shared(); check_if_ptr_null_return_value(tunnel, insufficient_resources); retcode rc = tunnel->attachbufferqueue(it->bufferqueue_); check_if_not_equal_return_value(rc, rc_ok, invalid_argument); if (stream->attachstreamtunnel(tunnel) != rc_ok) { camera_loge(attach buffer queue to stream [id = %{public}d] failed, it->streamid_); return invalid_argument; } } { std::lock_guard l(streamlock_); streammap_[stream->getstreamid()] = stream; // 保存流实例 }// ...} (左右移动查看全部内容)
从上面可以看出,消费端传递到了hal,那必然是由hal从bufferproducer获取buffer,并触发预览的启动流程。那看看attachstreamtunnel 的实现吧
c++retcode streambase::attachstreamtunnel(std::shared_ptr& tunnel){ if (state_ == stream_state_busy || state_ == stream_state_offline) { return rc_error; } tunnel_ = tunnel; // 绑定生产端 check_if_ptr_null_return_value(tunnel_, rc_error); tunnel_->setbuffercount(getbuffercount()); // 配置轮转的buffer个数 tunnelconfig config = {(uint32_t)streamconfig_.width, (uint32_t)streamconfig_.height, (uint32_t)streamconfig_.format, streamconfig_.usage}; tunnel_->config(config); streamconfig_.tunnelmode = true; return rc_ok;} (左右移动查看全部内容)
createstream之后便是commitstream,这里的commitstream 做了些什么事情呢,我们接着往下看
c++camretcode streamoperator::commitstreams(operationmode mode, const std::shared_ptr& modesetting){// ...... std::vector configs = {}; { std::lock_guard l(streamlock_); for (auto it : streammap_) { // 获取流的配置,前文createstrea时保存的流 configs.emplace_back(it.second->getstreamattribute()); } } // 检查流是否被支持 dynamicstreamswitchmode method = streampipeline_->checkstreamssupported(mode, modesetting, configs); if (method == dynamic_stream_switch_not_support) { return invalid_argument; } if (method == dynamic_stream_switch_need_inner_restart) { std::lock_guard l(streamlock_); for (auto it : streammap_) { it.second->stopstream();// 如果流被支持,但需要内部重启,这里先停流 } } { std::lock_guard l(streamlock_); for (auto it : streammap_) { if (it.second->commitstream() != rc_ok) { // 真正的 commitstream,下面再细说 camera_loge(commit stream [id = %{public}d] failed., it.first); return device_error; } } } retcode rc = streampipeline_->preconfig(modesetting); // 把模式传入进行预配置 if (rc != rc_ok) { camera_loge(prepare mode settings failed); return device_error; } rc = streampipeline_->createpipeline(mode);// 创建pipeline if (rc != rc_ok) { camera_loge(create pipeline failed.); return invalid_argument; } dfx_local_hitrace_end; return no_error;} (左右移动查看全部内容)
c++retcode streambase::commitstream(){// ... hoststreammgr_ = pipelinecore_->gethoststreammgr(); //从pipelinecore获取hoststreamanager check_if_ptr_null_return_value(hoststreammgr_, rc_error);// ... info.bufferpoolid_ = poolid_; info.buffercount_ = getbuffercount(); // 初始化 bufferpool retcode rc = bufferpool_->init(streamconfig_.width, streamconfig_.height, streamconfig_.usage, streamconfig_.format, getbuffercount(), camera_buffer_source_type_external); if (rc != rc_ok) { camera_loge(stream [id:%{public}d] initialize buffer pool failed., streamid_); return rc_error; } }// stream传递到pipelinecore 并进行绑定 retcode rc = hoststreammgr_->createhoststream(info, [this](std::shared_ptr buffer) { handleresult(buffer); return; });// .... return rc_ok;} (左右移动查看全部内容)
createstream 和commitstream结束之后便是capture,这里包含了起流的动作,关键实现如下
c++camretcode streamoperator::capture(int captureid, const std::shared_ptr& captureinfo, bool isstreaming){// ...// captureid 捕获请求的id; captureinfo 预览/拍照/录像的参数;isstreaming 连续捕获还是单次捕获(拍照) capturesetting setting = captureinfo->capturesetting_; auto request = std::make_shared(captureid, captureinfo->streamids_.size(), setting, captureinfo->enableshuttercallback_, isstreaming); for (auto id : captureinfo->streamids_) { // 创建捕获请求,并传递给前文创建的流 retcode rc = streammap_[id]->addrequest(request); if (rc != rc_ok) { return device_error; } }// ...} (左右移动查看全部内容)
从上面的代码可知预览、拍照、录像都是通过捕获请求触发,单次拍照则为单次捕获请求,预览和录像则是连续捕获请求。
c++retcode streambase::addrequest(std::shared_ptr& request){ check_if_ptr_null_return_value(request, rc_error); request->addowner(shared_from_this()); request->setfirstrequest(false); if (isfirstrequest) { retcode rc = startstream(); // 起流 if (rc != rc_ok) { camera_loge(start stream [id:%{public}d] failed, streamid_); return rc_error; } request->setfirstrequest(true); isfirstrequest = false; } { std::unique_lock l(wtlock_); waitinglist_.emplace_back(request); // 捕获请求添加到waitinglist cv_.notify_one(); } return rc_ok;} (左右移动查看全部内容)
看看streamstream是怎么实现的吧
c++retcode streambase::startstream(){// ... retcode rc = pipeline_->prepare({streamid_}); // pipeline先完成一些准备工作// ... state_ = stream_state_busy; std::string threadname = g_availablestreamtype[static_cast(streamtype_)] + # + std::to_string(streamid_); handler_ = std::make_unique([this, &threadname] {// 创建轮转线程 prctl(pr_set_name, threadname.c_str()); while (state_ == stream_state_busy) { handlerequest(); // 处理捕获请求 } });// ... rc = pipeline_->start({streamid_}); // 通知pipeline和底层硬件可以开始出帧了// ... return rc_ok;} (左右移动查看全部内容)
c++void streambase::handlerequest(){ // 如果有 捕获请求下发,则退出等待状态 if (waitinglist_.empty()) { std::unique_lock l(wtlock_); if (waitinglist_.empty()) { cv_.wait(l, [this] { return !(state_ == stream_state_busy && waitinglist_.empty()); }); } }// ... request = waitinglist_.front(); check_if_ptr_null_return_void(request); if (!request->iscontinous()) { // 如果是连续捕获,则保留一份拷贝在waitinglist waitinglist_.pop_front(); } }// 处理捕获请求 request->process(streamid_);// 最终调用下面的capture接口 return;} (左右移动查看全部内容)
c++retcode streambase::capture(const std::shared_ptr& request){ check_if_ptr_null_return_value(request, rc_error); check_if_ptr_null_return_value(pipeline_, rc_error); retcode rc = rc_error; if (request->isfirstone() && !request->iscontinous()) { uint32_t n = getbuffercount(); for (uint32_t i = 0; i needcancel()) {// 被取消的捕获则退出 camera_loge(streambase::capture stream [id:%{public}d] request->needcancel, streamid_); return rc_ok; } rc = pipeline_->config({streamid_}, request->getcapturesetting());// 通知pipeline配置 if (rc != rc_ok) { camera_loge(stream [id:%{public}d] config pipeline failed., streamid_); return rc_error; } rc = pipeline_->capture({streamid_}, request->getcaptureid());// 这里的capture指的是pipeline中的source node开始回buffer { std::unique_lock l(tslock_); intransitlist_.emplace_back(request);// 处理过的捕获请求存放在intransitlist } return rc_ok;} (左右移动查看全部内容)
到这起流的流程就结束了,pipeline回上来的帧通过onframe接口处理
c++retcode streambase::onframe(const std::shared_ptr& request){// ... bool isended = false; if (!request->iscontinous()) { isended = true; } else if (request->needcancel()) { isended = true; } { // intransitlist_ may has multiple copies of continious-capture request, we just need erase one of them. std::unique_lock l(tslock_); for (auto it = intransitlist_.begin(); it != intransitlist_.end(); it++) { if ((*it) == request) { intransitlist_.erase(it);// 已经回帧的请求,从intransitlist删除 break; } } if (isended) { // if this is the last request of capture, send captureendedmessage. auto it = std::find(intransitlist_.begin(), intransitlist_.end(), request); if (it == intransitlist_.end()) { std::shared_ptr endmessage = std::make_shared(streamid_, request->getcaptureid(), request->getendtime(), request->getownercount(), tunnel_->getframecount()); camera_logv(end of stream [%d], ready to send end message, capture id = %d, streamid_, request->getcaptureid()); messenger_->sendmessage(endmessage); pipeline_->cancelcapture({streamid_});// 如果此次捕获结束,则取消捕获 } } } receivebuffer(buffer);// 底层返回的buffer送还到生产端,最终帧数据送到消费端 return rc_ok;} (左右移动查看全部内容)
附录:
linux和android的关系 - 知乎 (zhihu.com)
hal subsystem | android open source project (google.cn)
zh-cn/release-notes/openharmony-v3.1-release.md · openharmony/docs - gitee.com
openharmony hdf 驱动框架介绍和驱动加载过程分析-openharmony技术社区
openharmony hdf hdi基础能力分析与使用
原文标题:openharmony 相机用户态驱动框架
文章出处:【微信公众号:harmonyos官方合作社区】欢迎添加关注!文章转载请注明出处。
不玩套路 创维董海涛京东直播引领直播新潮流
电气维修方法论第十七篇(电路的规格样式)
铜包铝电线的优缺点_铜包铝电线的功能
奥迪A4L将增1.4T车型,售价大幅降低,圆你豪车梦!
给大家介绍一下有关“地”的知识
详解HDI Implementation中的预览流程
索尼新一代主机卖点依然是图形性能
BT500助力蓄电池定期内阻检测
洛克希德以NEC的AI软件分析太空数据
理想汽车辅助驾驶发生碰撞,智能汽车的安全性再起争议
高性能电驱动系统NVH实验室工艺设计
你会选择搭载五摄的三星S10吗?
三星投资区块链?区块链初创公司Fliament500万美元A轮融资
如何在城市中推进千兆城市的建设?
三防平板电脑除了防水防摔还有哪些功能
展望成像系统的未来
2023工业机器人订单都去哪儿了?下半年押注什么?
ADC0809用法详解_引脚图及功能_工作原理_内部结构及应用电路
机器视觉检测能为生产制造企业带来哪些切实利益?
一加7T Pro将定位5G的手机市场,10月份发布,无奈价格太能打了!