https://android.googlesource.com/platform/dalvik.git
https://android.googlesource.com/platform/dalvik.git/+/master/docs/
http://elinux.org/Android_Dalvik_VM
http://en.wikipedia.org/wiki/Dalvik_(software)
Dalvik, Android’s virtual machine, generates significant debate
http://hi.baidu.com/seucrcr/item/2b988c570cb63c9208be1778
http://hllvm.group.iteye.com/group/topic/17798
这是一篇原来使用Core dump的记录，整理资料的时候看到的，没有深入的分析，只是用法
基本知识不清楚的话，请在网络上搜寻查阅
http://en.wikipedia.org/wiki/Core_dump
http://en.wikipedia.org/wiki/GNU_Debugger
guohai@KNIGHT:~$ ulimit -c 0 guohai@KNIGHT:~$ ulimit -c unlimited guohai@KNIGHT:~$ ulimit -c unlimited guohai@KNIGHT:~$ ./a.out Floating point exception (core dumped)
a.out是需要分析的程序，以上命令就是Linux上使用方法，很简单
更多情况请参考
http://www.cppblog.com/kongque/archive/2011/03/07/141262.aspx
那么在Android上怎么开启呢(首先得有root权限)？
$ adb remount $ adb shell root@android:/ # ulimit -c unlimited
更改Core dump档案存储的路径(这个存储的路径可以根据需要定制)
root@android:/ # echo "/data/coredump/%p_%e" > /proc/sys/kernel/core_pattern
这样当有native crash存在的时候就会出现对应的Core dump档案了
然后就把档案拷贝到宿主机上，用GDB去载入档案，分析出错的原因
———–EOF———–
说到架构/构架就觉得心虚，感觉这两个字太高端了，所以我只敢用结构来表达。
本身我也不是做HAL或者Driver的，我只是写APP的。
出于码农习惯，我会尝试去了解一些比自己用到的深入一点的东西，也比较想向系统这一层开发迈进(哪位高手收了我吧)
以下表述如果存在错误，请指教！
之前有在一篇博客中提到Camera相关的东西
http://guoh.org/lifelog/2012/09/asop-camera-source-code-reading-video-snapshot/
比如
$ANDROID_SRC_HOME/frameworks/base/core/java/android/hardware/Camera.java $ANDROID_SRC_HOME/frameworks/base/core/jni/android_hardware_Camera.cpp
// 注：目前以下这两个文件在JB当中应该被移动位置了，由此可见Android也是个活跃的项目
$ANDROID_SRC_HOME/frameworks/base/include/camera/Camera.h $ANDROID_SRC_HOME/frameworks/base/libs/camera/Camera.cpp
这篇博客是阅读http://source.android.com/devices/camera.html，结合在JB代码的基础上写的笔记，可能不是那么详细和有逻辑，只关注重点和要注意的地方，所以如果你刚好读到，但是你又没有什么Android Camera的基础的话，建议还是先看看http://developer.android.com/guide/topics/media/camera.html和Reference
正式开始
大体来说Camera分为这么多部分
APP — JNI — NATIVE — BINDER — CAMERA SERVICE — HAL — DRIVER
下面列出这每一部分对应于AOSP当中的代码
APP($ANDROID_SRC_HOME/frameworks/base/core/java/android/hardware/Camera.java)
JNI($ANDROID_SRC_HOME/frameworks/base/core/jni/android_hardware_Camera.cpp)
NATIVE($ANDROID_SRC_HOME/frameworks/av/camera/Camera.cpp)
BINDER($ANDROID_SRC_HOME/frameworks/av/camera/ICameraService.cpp
$ANDROID_SRC_HOME/frameworks/av/camera/ICameraClient.cpp)
CAMERA SERVICE($ANDROID_SRC_HOME/frameworks/av/services/camera/libcameraservice/CameraService.cpp)
HAL($ANDROID_SRC_HOME/hardware/libhardware/include/hardware/camera.h
$ANDROID_SRC_HOME/hardware/libhardware/include/hardware/camera_common.h
$ANDROID_SRC_HOME/frameworks/av/services/camera/libcameraservice/CameraHardwareInterface.h)
对于Galaxy Nexus来讲，HAL的具体实现位于
$ANDROID_SRC_HOME/hardware/ti/omap4xxx/camera/CameraHal.cpp $ANDROID_SRC_HOME/hardware/ti/omap4xxx/camera/CameraHalCommon.cpp
头文件位于
$ANDROID_SRC_HOME/hardware/ti/omap4xxx/camera/inc/
对于Galaxy Nexus来讲
DRIVER(https://developers.google.com/android/nexus/drivers)
不知道看了对Camera有没有个大体的认识？
大致了解了的话，就翻代码出来看看自己感兴趣的部分吧
你可以通过repo拉取所有代码到本地看，也可以在线查看。
android/hardware/Camera.java
public static Camera open(int cameraId) {
return new Camera(cameraId);
}
Camera(int cameraId) {
mShutterCallback = null;
mRawImageCallback = null;
mJpegCallback = null;
mPreviewCallback = null;
mPostviewCallback = null;
mZoomListener = null;
Looper looper;
if ((looper = Looper.myLooper()) != null) {
mEventHandler = new EventHandler(this, looper);
} else if ((looper = Looper.getMainLooper()) != null) {
mEventHandler = new EventHandler(this, looper);
} else {
mEventHandler = null;
}
native_setup(new WeakReference<Camera>(this), cameraId);
}
以上代码描述了我们所要研究的东西的基础，开启camera，跟所有的硬件一样，使用之前需要先开启。
这里有个需要注意的地方就是mEventHandler，这个变量是做什么用的，文档有解释的比较清楚，如下：
Callbacks from other methods are delivered to the event loop of the
thread which called open(). If this thread has no event loop, then
callbacks are delivered to the main application event loop. If there
is no main application event loop, callbacks are not delivered.
如果你不是写一个玩具程序的话，你就应当注意这里的描述。
随着native_setup(new WeakReference
android_hardware_Camera.cpp
// connect to camera service
static void android_hardware_Camera_native_setup(JNIEnv *env, jobject thiz,
jobject weak_this, jint cameraId)
{
sp<Camera> camera = Camera::connect(cameraId);
if (camera == NULL) {
jniThrowRuntimeException(env, "Fail to connect to camera service");
return;
}
// make sure camera hardware is alive
if (camera->getStatus() != NO_ERROR) {
jniThrowRuntimeException(env, "Camera initialization failed");
return;
}
jclass clazz = env->GetObjectClass(thiz);
if (clazz == NULL) {
jniThrowRuntimeException(env, "Can't find android/hardware/Camera");
return;
}
// We use a weak reference so the Camera object can be garbage collected.
// The reference is only used as a proxy for callbacks.
sp<JNICameraContext> context = new JNICameraContext(env, weak_this, clazz, camera);
context->incStrong(thiz);
camera->setListener(context);
// save context in opaque field
env->SetIntField(thiz, fields.context, (int)context.get());
}
随着sp
av/include/camera/Camera.h，当然实现都在av/camera/Camera.cpp
sp<Camera> Camera::connect(int cameraId)
{
ALOGV("connect");
sp<Camera> c = new Camera();
const sp<ICameraService>& cs = getCameraService();
if (cs != 0) {
c->mCamera = cs->connect(c, cameraId);
}
if (c->mCamera != 0) {
c->mCamera->asBinder()->linkToDeath(c);
c->mStatus = NO_ERROR;
} else {
c.clear();
}
return c;
}
这里getCameraService()是一个比较关键的东西，需要仔细看看。
// establish binder interface to camera service
const sp<ICameraService>& Camera::getCameraService()
{
Mutex::Autolock _l(mLock);
if (mCameraService.get() == 0) {
sp<IServiceManager> sm = defaultServiceManager();
sp<IBinder> binder;
do {
binder = sm->getService(String16("media.camera"));
if (binder != 0)
break;
ALOGW("CameraService not published, waiting...");
usleep(500000); // 0.5 s
} while(true);
if (mDeathNotifier == NULL) {
mDeathNotifier = new DeathNotifier();
}
binder->linkToDeath(mDeathNotifier);
mCameraService = interface_cast<ICameraService>(binder);
}
ALOGE_IF(mCameraService==0, "no CameraService!?");
return mCameraService;
}
然后就是av/camera/ICameraService.cpp当中代理类BpCameraService的
// connect to camera service
virtual sp<ICamera> connect(const sp<ICameraClient>& cameraClient, int cameraId)
{
Parcel data, reply;
data.writeInterfaceToken(ICameraService::getInterfaceDescriptor());
data.writeStrongBinder(cameraClient->asBinder());
data.writeInt32(cameraId);
remote()->transact(BnCameraService::CONNECT, data, &reply);
return interface_cast<ICamera>(reply.readStrongBinder());
}
最终转移到BnCameraService的connect方法，但是这是个纯虚函数，而且CameraService继承了BnCameraService，所以请看
av/services/camera/libcameraservice/CameraService.h，当然具体实现还是在av/services/camera/libcameraservice/CameraService.cpp当中。
sp<ICamera> CameraService::connect(
const sp<ICameraClient>& cameraClient, int cameraId) {
int callingPid = getCallingPid();
LOG1("CameraService::connect E (pid %d, id %d)", callingPid, cameraId);
if (!mModule) {
ALOGE("Camera HAL module not loaded");
return NULL;
}
sp<Client> client;
if (cameraId < 0 || cameraId >= mNumberOfCameras) {
ALOGE("CameraService::connect X (pid %d) rejected (invalid cameraId %d).",
callingPid, cameraId);
return NULL;
}
char value[PROPERTY_VALUE_MAX];
property_get("sys.secpolicy.camera.disabled", value, "0");
if (strcmp(value, "1") == 0) {
// Camera is disabled by DevicePolicyManager.
ALOGI("Camera is disabled. connect X (pid %d) rejected", callingPid);
return NULL;
}
Mutex::Autolock lock(mServiceLock);
if (mClient[cameraId] != 0) {
client = mClient[cameraId].promote();
if (client != 0) {
if (cameraClient->asBinder() == client->getCameraClient()->asBinder()) {
LOG1("CameraService::connect X (pid %d) (the same client)",
callingPid);
return client;
} else {
ALOGW("CameraService::connect X (pid %d) rejected (existing client).",
callingPid);
return NULL;
}
}
mClient[cameraId].clear();
}
if (mBusy[cameraId]) {
ALOGW("CameraService::connect X (pid %d) rejected"
" (camera %d is still busy).", callingPid, cameraId);
return NULL;
}
struct camera_info info;
if (mModule->get_camera_info(cameraId, &info) != OK) {
ALOGE("Invalid camera id %d", cameraId);
return NULL;
}
int deviceVersion;
if (mModule->common.module_api_version == CAMERA_MODULE_API_VERSION_2_0) {
deviceVersion = info.device_version;
} else {
deviceVersion = CAMERA_DEVICE_API_VERSION_1_0;
}
switch(deviceVersion) {
case CAMERA_DEVICE_API_VERSION_1_0:
client = new CameraClient(this, cameraClient, cameraId,
info.facing, callingPid, getpid());
break;
case CAMERA_DEVICE_API_VERSION_2_0:
client = new Camera2Client(this, cameraClient, cameraId,
info.facing, callingPid, getpid());
break;
default:
ALOGE("Unknown camera device HAL version: %d", deviceVersion);
return NULL;
}
if (client->initialize(mModule) != OK) {
return NULL;
}
cameraClient->asBinder()->linkToDeath(this);
mClient[cameraId] = client;
LOG1("CameraService::connect X (id %d, this pid is %d)", cameraId, getpid());
return client;
}
比如我们这里以HAL 1.0来看，参见
av/services/camera/libcameraservice/CameraClient.h
和
av/services/camera/libcameraservice/CameraClient.cpp
CameraClient::CameraClient(const sp<CameraService>& cameraService,
const sp<ICameraClient>& cameraClient,
int cameraId, int cameraFacing, int clientPid, int servicePid):
Client(cameraService, cameraClient,
cameraId, cameraFacing, clientPid, servicePid)
{
int callingPid = getCallingPid();
LOG1("CameraClient::CameraClient E (pid %d, id %d)", callingPid, cameraId);
mHardware = NULL;
mMsgEnabled = 0;
mSurface = 0;
mPreviewWindow = 0;
mDestructionStarted = false;
// Callback is disabled by default
mPreviewCallbackFlag = CAMERA_FRAME_CALLBACK_FLAG_NOOP;
mOrientation = getOrientation(0, mCameraFacing == CAMERA_FACING_FRONT);
mPlayShutterSound = true;
LOG1("CameraClient::CameraClient X (pid %d, id %d)", callingPid, cameraId);
}
status_t CameraClient::initialize(camera_module_t *module) {
int callingPid = getCallingPid();
LOG1("CameraClient::initialize E (pid %d, id %d)", callingPid, mCameraId);
char camera_device_name[10];
status_t res;
snprintf(camera_device_name, sizeof(camera_device_name), "%d", mCameraId);
mHardware = new CameraHardwareInterface(camera_device_name);
res = mHardware->initialize(&module->common);
if (res != OK) {
ALOGE("%s: Camera %d: unable to initialize device: %s (%d)",
__FUNCTION__, mCameraId, strerror(-res), res);
mHardware.clear();
return NO_INIT;
}
mHardware->setCallbacks(notifyCallback,
dataCallback,
dataCallbackTimestamp,
(void *)mCameraId);
// Enable zoom, error, focus, and metadata messages by default
enableMsgType(CAMERA_MSG_ERROR | CAMERA_MSG_ZOOM | CAMERA_MSG_FOCUS |
CAMERA_MSG_PREVIEW_METADATA | CAMERA_MSG_FOCUS_MOVE);
LOG1("CameraClient::initialize X (pid %d, id %d)", callingPid, mCameraId);
return OK;
}
当然还会用到
libhardware//include/hardware/camera.h
和
libhardware//include/hardware/camera_common.h
和
av/services/camera/libcameraservice/CameraHardwareInterface.h
status_t initialize(hw_module_t *module)
{
ALOGI("Opening camera %s", mName.string());
int rc = module->methods->open(module, mName.string(),
(hw_device_t **)&mDevice);
if (rc != OK) {
ALOGE("Could not open camera %s: %d", mName.string(), rc);
return rc;
}
initHalPreviewWindow();
return rc;
}
这只是一个接口，具体的实现是厂商来做的啦
以上是从上层调用到底层，接着我们还会看一个从底层调用到上层的情况。
从代码看出有绑定3个callback到driver
mHardware->setCallbacks(notifyCallback,
dataCallback,
dataCallbackTimestamp,
(void *)mCameraId);
所以这三个callback会有机会被call到
以dataCallback为例子，其中有一个是
// picture callback - compressed picture ready
void CameraClient::handleCompressedPicture(const sp<IMemory>& mem) {
disableMsgType(CAMERA_MSG_COMPRESSED_IMAGE);
sp<ICameraClient> c = mCameraClient;
mLock.unlock();
if (c != 0) {
c->dataCallback(CAMERA_MSG_COMPRESSED_IMAGE, mem, NULL);
}
}
通过ICameraClient调用会app，这里的ICameraClient是BpCameraClient(为什么是Bp，后面会稍加解释)
// generic data callback from camera service to app with image data
void BpCameraClient::dataCallback(int32_t msgType, const sp<IMemory>& imageData,
camera_frame_metadata_t *metadata)
{
ALOGV("dataCallback");
Parcel data, reply;
data.writeInterfaceToken(ICameraClient::getInterfaceDescriptor());
data.writeInt32(msgType);
data.writeStrongBinder(imageData->asBinder());
if (metadata) {
data.writeInt32(metadata->number_of_faces);
data.write(metadata->faces, sizeof(camera_face_t) * metadata->number_of_faces);
}
remote()->transact(DATA_CALLBACK, data, &reply, IBinder::FLAG_ONEWAY);
}
于是
status_t BnCameraClient::onTransact(
uint32_t code, const Parcel& data, Parcel* reply, uint32_t flags)
{
switch(code) {
case NOTIFY_CALLBACK: {
...
} break;
case DATA_CALLBACK: {
ALOGV("DATA_CALLBACK");
CHECK_INTERFACE(ICameraClient, data, reply);
int32_t msgType = data.readInt32();
sp<IMemory> imageData = interface_cast<IMemory>(data.readStrongBinder());
camera_frame_metadata_t *metadata = NULL;
if (data.dataAvail() > 0) {
metadata = new camera_frame_metadata_t;
metadata->number_of_faces = data.readInt32();
metadata->faces = (camera_face_t *) data.readInplace(
sizeof(camera_face_t) * metadata->number_of_faces);
}
dataCallback(msgType, imageData, metadata); // 因为pure virtual function的关系，所以会call到Camera当中的实现
if (metadata) delete metadata;
return NO_ERROR;
} break;
case DATA_CALLBACK_TIMESTAMP: {
...
} break;
default:
return BBinder::onTransact(code, data, reply, flags);
}
}
于是
// callback from camera service when frame or image is ready
void Camera::dataCallback(int32_t msgType, const sp<IMemory>& dataPtr,
camera_frame_metadata_t *metadata)
{
sp<CameraListener> listener;
{
Mutex::Autolock _l(mLock);
listener = mListener;
}
if (listener != NULL) {
listener->postData(msgType, dataPtr, metadata);
}
}
从app call到service是BnCameraClient(client是实体，service是代理，从new的代码就可以观察出来)
从service call到app是BpCameraClient(service是实体，client是代理)
而这个Bn到Bp的转换是通过
sp<ICameraClient> cameraClient = interface_cast<ICameraClient>(data.readStrongBinder());
完成的。
后面我们会列举些需要注意的点：
ICameraClient继承自IInterface
BnCameraClient继承自BnInterface
Camera继承自BnCameraClient
ICameraService继承自IInterface
BnCameraService继承自BnInterface
ICamera继承自IInterface
BnCamera继承自BnInterface
CameraService继承自BinderService，BnCameraService，IBinder::DeathRecipient
而BinderService主要是用来实例化service的统一接口
CameraService有个嵌套类Client继承自BnCamera
真正的实现大部分在CameraClient当中，现在应该是Camera2Client
这才是真正的client，直接调用HW或者被HW调用。
sendCommand往driver送参数的时候会先检验是不是CameraService这层可以处理的东西(有些东西不需要到driver的)
cmd == CAMERA_CMD_SET_DISPLAY_ORIENTATION
cmd == CAMERA_CMD_ENABLE_SHUTTER_SOUND
cmd == CAMERA_CMD_PLAY_RECORDING_SOUND
cmd == CAMERA_CMD_SET_VIDEO_BUFFER_COUNT
cmd == CAMERA_CMD_PING
以上这些都CameraService自己处理了
其他均丢给driver(mHardware->sendCommand(cmd, arg1, arg2);)
接下来是一些疑问点：
// these are initialized in the constructor.
sp
mHardware是在哪里被清除的？
// Make sure disconnect() is done once and once only, whether it is called
// from the user directly, or called by the destructor.
if (mHardware == 0) return;
所以这个判断会走到？
内容均来自于网络资料，源码以及自己的理解，如有错误的地方还请指出！示例源码可以随意使用。
我这里使用的环境如下
PLATFORM_VERSION_CODENAME=AOSP PLATFORM_VERSION=4.0.9.99.999.9999.99999 TARGET_PRODUCT=full_panda TARGET_BUILD_VARIANT=userdebug TARGET_BUILD_TYPE=release
Android提供了一些基础服务，但是如果你想给它添加一些特定的服务的话，也是可以的(比如在一直到某些特定的平台下，需要增加/裁剪一些服务)
比如我们都知道Android上用的最广泛的应该是多媒体服务，其对应的有一个mediaserver进程，然后比如我们的
AudioFlinger/MediaPlayerService/CameraService/AudioPolicyService
都是跑在这个进程里面的
那如果我们想自己增加一个类似的东西可不可以呢？答案当然是可以的。
很简单的几点就可以帮你完成
启动进程
/path/to/aosp/frameworks/base/customized_service/server
具体服务实现
/path/to/aosp/frameworks/base/customized_service/libcustomizedservice
系统开机启动
/path/to/aosp/system/core/rootdir/init.rc
添加如下内容到init.rc文件
service customized /system/bin/customizedserver
class main
user media
group audio camera
我是用的Pandaboard
编译整个ROM，然后fastboot flashall
I/ ( 1080): ServiceManager: 0x40ce4f00
D/ ( 1080): CustomizedService instantiate
D/ ( 1080): CustomizedService created
E/ServiceManager( 94): add_service('test.customized',0x44) uid=xxxx - PERMISSION DENIED
D/ ( 1080): CustomizedService r = -1
D/ ( 1080): CustomizedService destroyed
或者你想从shell当中启动这个服务的进程也会出现类似权限不足的问题，追一下log就很快会发现问题
int do_add_service(struct binder_state *bs,
uint16_t *s, unsigned len,
void *ptr, unsigned uid, int allow_isolated)
{
struct svcinfo *si;
if (!ptr || (len == 0) || (len > 127))
return -1;
if (!svc_can_register(uid, s)) {
ALOGE("add_service('%s',%p) uid=%d - PERMISSION DENIED\n",
str8(s), ptr, uid);
return -1;
}
si = find_svc(s, len);
......
binder_acquire(bs, ptr);
binder_link_to_death(bs, ptr, &si->death);
return 0;
}
int svc_can_register(unsigned uid, uint16_t *name)
{
unsigned n;
if ((uid == 0) || (uid == AID_SYSTEM))
return 1;
for (n = 0; n < sizeof(allowed) / sizeof(allowed[0]); n++)
if ((uid == allowed[n].uid) && str16eq(name, allowed[n].name))
return 1;
return 0;
}
可以看出能注册service的用户的uid只能是0即ROOT或者是AID_SYSTEM，或者是allowed列表当中的服务，最简单的就是把所要添加的服务加到列表当中
{ AID_MEDIA, "test.customized" },
而服务allowed列表位于
/path/to/aosp/frameworks/base/cmds/servicemanager/service_manager.c
重新编译ROM，烧完之后启动看到customizedservice这个进程起来的话，就多半表明服务加好了
App端怎么调用？
写个Activity利用android.os.ServiceManager的getService获取到指定的服务，然后调用就可以了。
这里需要说明下的就是目前android.os.ServiceManager对于app来说是hide掉的，我们暂且可以不管，用个反射先用着。
至于为什么要hide掉，难道Android不想让我们自己定义系统级别的service来使用？还是说用其他的方式可以完成，目前我也不清楚！
目前结果就是通过getService能获取到我们的服务，并且成功调用，但是因为我返回的参数有String有int，而String没有获取出来，都知道Parcel的参数顺序是很重要的，可能是这个的原因，后面再来看看到底是什么原因，目前就是把customized service这一套先跑通。
P.S. 后来一次发现可能是我每次编译的so档案没有进到ROM当中去，所以烧到设备的so档案不是最新的，就没有把后面加的String获取出来(之前是两个int型数据)。
另外我还做了个实验，就是在service端通过ServiceManager的getService(…)方法再次访问当前service，这个就相当于在当前代码中通过一个指针或者引用访问的效果是一样的，这就是Binder的强大之处吧！此改动可以参见以下commit：
commit 1875ad41d5f8e505cb4c26599e4626944005f26b
Date: Tue Jun 25 12:10:59 2013 +0800Symptom: test invoking getService in same process with service
Issue ID: NA
Root Cause: NA
Solution: NA
完整代码参见
https://github.com/guohai/and-customized-service.git
这里是Java端调用native端的服务，其实也可以native调用native的服务，这里暂时就不再去具体实现了，需要的话，会再来写。
无图无真相
前面有分析过Camera的实现，现在来看看MediaRecorder的实现，这里我不会太去关注它的分层结构，我更关注它的逻辑！
APP层 /path/to/aosp/frameworks/base/media/java/android/media/MediaRecorder.java
JNI层 /path/to/aosp/frameworks/base/media/jni/android_media_MediaRecorder.cpp
调用NATIVE层的MediaRecorder(这里是BnMediaRecorderClient)
header /path/to/aosp/frameworks/av/include/media/mediarecorder.h
implementation /path/to/aosp/frameworks/av/media/libmedia/mediarecorder.cpp
MediaRecorder::MediaRecorder() : mSurfaceMediaSource(NULL)
{
ALOGV("constructor");
const sp<IMediaPlayerService>& service(getMediaPlayerService());
if (service != NULL) {
mMediaRecorder = service->createMediaRecorder(getpid());
}
if (mMediaRecorder != NULL) {
mCurrentState = MEDIA_RECORDER_IDLE;
}
doCleanUp();
}
getMediaPlayerService()这个方法位于/path/to/aosp/frameworks/av/include/media/IMediaDeathNotifier.h
获取到MediaPlayerService(这个是BpMediaPlayerService)之后
调用IMediaPlayerService当中的
sp<IMediaRecorder> MediaPlayerService::createMediaRecorder(pid_t pid)
{
sp<MediaRecorderClient> recorder = new MediaRecorderClient(this, pid);
wp<MediaRecorderClient> w = recorder;
Mutex::Autolock lock(mLock);
mMediaRecorderClients.add(w);
ALOGV("Create new media recorder client from pid %d", pid);
return recorder;
}
创建MediaRecorderClient(这里是BnMediaRecorder)
但是通过binder拿到的是BpMediaRecorder
因为有如下的interface_cast过程
virtual sp<IMediaRecorder> createMediaRecorder(pid_t pid)
{
Parcel data, reply;
data.writeInterfaceToken(IMediaPlayerService::getInterfaceDescriptor());
data.writeInt32(pid);
remote()->transact(CREATE_MEDIA_RECORDER, data, &reply);
return interface_cast<IMediaRecorder>(reply.readStrongBinder());
}
而MediaRecorderClient当中又会创建StagefrightRecorder(MediaRecorderBase)，它位于
/path/to/aosp/frameworks/av/media/libmediaplayerservice/StagefrightRecorder.cpp
目前我们可以认为在APP/JNI/NATIVE这边是在一个进程当中，在MediaPlayerService当中的MediaRecorderClient/StagefrightRecorder是在另外一个进程当中，他们之间通过binder通信，而且Bp和Bn我们也都有拿到，后面我们将不再仔细区分Bp和Bn。
客户端这边
BnMediaRecorderClient
BpMediaRecorder
BpMediaPlayerService服务端这边
BpMediaRecorderClient(如果需要通知客户端的话，它可以获得这个Bp)
BnMediaRecorder
BnMediaPlayerService
我们以开始录影为例子，比如start()
在这里就兵分两路，一个CameraSource，一个MPEG4Writer(sp<MediaWriter> mWriter)
这两个class都位于/path/to/aosp/frameworks/av/media/libstagefright/当中
status_t StagefrightRecorder::startMPEG4Recording() {
int32_t totalBitRate;
status_t err = setupMPEG4Recording(
mOutputFd, mVideoWidth, mVideoHeight,
mVideoBitRate, &totalBitRate, &mWriter);
if (err != OK) {
return err;
}
int64_t startTimeUs = systemTime() / 1000;
sp<MetaData> meta = new MetaData;
setupMPEG4MetaData(startTimeUs, totalBitRate, &meta);
err = mWriter->start(meta.get());
if (err != OK) {
return err;
}
return OK;
}
status_t StagefrightRecorder::setupMPEG4Recording(
int outputFd,
int32_t videoWidth, int32_t videoHeight,
int32_t videoBitRate,
int32_t *totalBitRate,
sp<MediaWriter> *mediaWriter) {
mediaWriter->clear();
*totalBitRate = 0;
status_t err = OK;
sp<MediaWriter> writer = new MPEG4Writer(outputFd);
if (mVideoSource < VIDEO_SOURCE_LIST_END) {
sp<MediaSource> mediaSource;
err = setupMediaSource(&mediaSource); // very important
if (err != OK) {
return err;
}
sp<MediaSource> encoder;
err = setupVideoEncoder(mediaSource, videoBitRate, &encoder); // very important
if (err != OK) {
return err;
}
writer->addSource(encoder);
*totalBitRate += videoBitRate;
}
// Audio source is added at the end if it exists.
// This help make sure that the "recoding" sound is suppressed for
// camcorder applications in the recorded files.
if (!mCaptureTimeLapse && (mAudioSource != AUDIO_SOURCE_CNT)) {
err = setupAudioEncoder(writer); // very important
if (err != OK) return err;
*totalBitRate += mAudioBitRate;
}
...
writer->setListener(mListener);
*mediaWriter = writer;
return OK;
}
// Set up the appropriate MediaSource depending on the chosen option
status_t StagefrightRecorder::setupMediaSource(
sp<MediaSource> *mediaSource) {
if (mVideoSource == VIDEO_SOURCE_DEFAULT
|| mVideoSource == VIDEO_SOURCE_CAMERA) {
sp<CameraSource> cameraSource;
status_t err = setupCameraSource(&cameraSource);
if (err != OK) {
return err;
}
*mediaSource = cameraSource;
} else if (mVideoSource == VIDEO_SOURCE_GRALLOC_BUFFER) {
// If using GRAlloc buffers, setup surfacemediasource.
// Later a handle to that will be passed
// to the client side when queried
status_t err = setupSurfaceMediaSource();
if (err != OK) {
return err;
}
*mediaSource = mSurfaceMediaSource;
} else {
return INVALID_OPERATION;
}
return OK;
}
status_t StagefrightRecorder::setupCameraSource(
sp<CameraSource> *cameraSource) {
status_t err = OK;
if ((err = checkVideoEncoderCapabilities()) != OK) {
return err;
}
Size videoSize;
videoSize.width = mVideoWidth;
videoSize.height = mVideoHeight;
if (mCaptureTimeLapse) {
if (mTimeBetweenTimeLapseFrameCaptureUs < 0) {
ALOGE("Invalid mTimeBetweenTimeLapseFrameCaptureUs value: %lld",
mTimeBetweenTimeLapseFrameCaptureUs);
return BAD_VALUE;
}
mCameraSourceTimeLapse = CameraSourceTimeLapse::CreateFromCamera(
mCamera, mCameraProxy, mCameraId,
videoSize, mFrameRate, mPreviewSurface,
mTimeBetweenTimeLapseFrameCaptureUs);
*cameraSource = mCameraSourceTimeLapse;
} else {
*cameraSource = CameraSource::CreateFromCamera(
mCamera, mCameraProxy, mCameraId, videoSize, mFrameRate,
mPreviewSurface, true /*storeMetaDataInVideoBuffers*/);
}
mCamera.clear();
mCameraProxy.clear();
if (*cameraSource == NULL) {
return UNKNOWN_ERROR;
}
if ((*cameraSource)->initCheck() != OK) {
(*cameraSource).clear();
*cameraSource = NULL;
return NO_INIT;
}
// When frame rate is not set, the actual frame rate will be set to
// the current frame rate being used.
if (mFrameRate == -1) {
int32_t frameRate = 0;
CHECK ((*cameraSource)->getFormat()->findInt32(
kKeyFrameRate, &frameRate));
ALOGI("Frame rate is not explicitly set. Use the current frame "
"rate (%d fps)", frameRate);
mFrameRate = frameRate;
}
CHECK(mFrameRate != -1);
mIsMetaDataStoredInVideoBuffers =
(*cameraSource)->isMetaDataStoredInVideoBuffers();
return OK;
}
status_t StagefrightRecorder::setupVideoEncoder(
sp<MediaSource> cameraSource,
int32_t videoBitRate,
sp<MediaSource> *source) {
source->clear();
sp<MetaData> enc_meta = new MetaData;
enc_meta->setInt32(kKeyBitRate, videoBitRate);
enc_meta->setInt32(kKeyFrameRate, mFrameRate);
switch (mVideoEncoder) {
case VIDEO_ENCODER_H263:
enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_H263);
break;
case VIDEO_ENCODER_MPEG_4_SP:
enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_MPEG4);
break;
case VIDEO_ENCODER_H264:
enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_AVC);
break;
default:
CHECK(!"Should not be here, unsupported video encoding.");
break;
}
sp<MetaData> meta = cameraSource->getFormat();
int32_t width, height, stride, sliceHeight, colorFormat;
CHECK(meta->findInt32(kKeyWidth, &width));
CHECK(meta->findInt32(kKeyHeight, &height));
CHECK(meta->findInt32(kKeyStride, &stride));
CHECK(meta->findInt32(kKeySliceHeight, &sliceHeight));
CHECK(meta->findInt32(kKeyColorFormat, &colorFormat));
enc_meta->setInt32(kKeyWidth, width);
enc_meta->setInt32(kKeyHeight, height);
enc_meta->setInt32(kKeyIFramesInterval, mIFramesIntervalSec);
enc_meta->setInt32(kKeyStride, stride);
enc_meta->setInt32(kKeySliceHeight, sliceHeight);
enc_meta->setInt32(kKeyColorFormat, colorFormat);
if (mVideoTimeScale > 0) {
enc_meta->setInt32(kKeyTimeScale, mVideoTimeScale);
}
if (mVideoEncoderProfile != -1) {
enc_meta->setInt32(kKeyVideoProfile, mVideoEncoderProfile);
}
if (mVideoEncoderLevel != -1) {
enc_meta->setInt32(kKeyVideoLevel, mVideoEncoderLevel);
}
OMXClient client;
CHECK_EQ(client.connect(), (status_t)OK);
uint32_t encoder_flags = 0;
if (mIsMetaDataStoredInVideoBuffers) {
encoder_flags |= OMXCodec::kStoreMetaDataInVideoBuffers;
}
// Do not wait for all the input buffers to become available.
// This give timelapse video recording faster response in
// receiving output from video encoder component.
if (mCaptureTimeLapse) {
encoder_flags |= OMXCodec::kOnlySubmitOneInputBufferAtOneTime;
}
sp<MediaSource> encoder = OMXCodec::Create(
client.interface(), enc_meta,
true /* createEncoder */, cameraSource,
NULL, encoder_flags);
if (encoder == NULL) {
ALOGW("Failed to create the encoder");
// When the encoder fails to be created, we need
// release the camera source due to the camera's lock
// and unlock mechanism.
cameraSource->stop();
return UNKNOWN_ERROR;
}
*source = encoder;
return OK;
}
这里和OMXCodec关联起来
有一个叫media_codecs.xml的配置文件来表明设备支持哪些codec
我们录制MPEG 4的时候还会有声音，所以后面还有个setupAudioEncoder，具体的方法就不展开了，总之就是把声音也作为一个Track加入到MPEG4Writer当中去。
这里插个题外话，Google说把setupAudioEncoder放到后面是为了避免开始录影的那一个提示声音也被录制进去，但是实际发现它这样做还是会有bug，在一些设备上还是会把那声录制进去，这个遇到的都是靠APP自己来播放声音来绕过这个问题的。
另外MPEG4Writer当中有个
start(MetaData*)
启动两个方法
a) startWriterThread
启动一个thread去写
void MPEG4Writer::threadFunc() {
ALOGV("threadFunc");
prctl(PR_SET_NAME, (unsigned long)"MPEG4Writer", 0, 0, 0);
Mutex::Autolock autoLock(mLock);
while (!mDone) {
Chunk chunk;
bool chunkFound = false;
while (!mDone && !(chunkFound = findChunkToWrite(&chunk))) {
mChunkReadyCondition.wait(mLock);
}
// Actual write without holding the lock in order to
// reduce the blocking time for media track threads.
if (chunkFound) {
mLock.unlock();
writeChunkToFile(&chunk);
mLock.lock();
}
}
writeAllChunks();
}
b) startTracks
status_t MPEG4Writer::startTracks(MetaData *params) {
for (List<Track *>::iterator it = mTracks.begin();
it != mTracks.end(); ++it) {
status_t err = (*it)->start(params);
if (err != OK) {
for (List<Track *>::iterator it2 = mTracks.begin();
it2 != it; ++it2) {
(*it2)->stop();
}
return err;
}
}
return OK;
}
然后调用每个Track的start方法
status_t MPEG4Writer::Track::start(MetaData *params) {
...
initTrackingProgressStatus(params);
...
status_t err = mSource->start(meta.get()); // 这里会去执行CameraSource(start)，这两个是相互关联的
...
pthread_create(&mThread, &attr, ThreadWrapper, this);
return OK;
}
void *MPEG4Writer::Track::ThreadWrapper(void *me) {
Track *track = static_cast<Track *>(me);
status_t err = track->threadEntry();
return (void *) err;
}
通过status_t MPEG4Writer::Track::threadEntry()
是新启动另外一个thread，它里面会通过一个循环来不断读取CameraSource(read)里面的数据，CameraSource里面的数据当然是从driver返回过来的(可以参见CameraSourceListener)，然后写到文件当中。
注意：准确来说这里MPEG4Writer读取的是OMXCodec里的数据，因为数据先到CameraSource，codec负责编码之后，MPEG4Writer才负责写到文件当中！关于数据在CameraSource/OMXCodec/MPEG4Writer之间是怎么传递的，可以参见http://guoh.org/lifelog/2013/06/interaction-between-stagefright-and-codec/当中讲Buffer的传输过程。
回头再来看，Stagefright做了什么事情？我更觉得它只是一个粘合剂(glue)的用处，它工作在MediaPlayerService这一层，把MediaSource，MediaWriter，Codec以及上层的MediaRecorder绑定在一起，这应该就是它最大的作用，Google用它来替换Opencore也是符合其一贯的工程派作风(相比复杂的学术派而言，虽然Google很多东西也很复杂，但是它一般都是以尽量简单的方式来解决问题)。
让大家觉得有点不习惯的是，它把MediaRecorder放在MediaPlayerService当中，这两个看起来是对立的事情，或者某一天它们会改名字，或者是两者分开，不知道～～
当然这只是个简单的大体介绍，Codec相关的后面争取专门来分析一下！
有些细节的东西在这里没有列出，需要的话会把一些注意点列出来：
1. 时光流逝录影
CameraSource对应的就是CameraSourceTimeLapse
具体做法就是在
dataCallbackTimestamp
当中有skipCurrentFrame
当然它是用些变量来记录和计算
mTimeBetweenTimeLapseVideoFramesUs(1E6/videoFrameRate) // 两个frame之间的间隔时间
记录上一个frame的(mLastTimeLapseFrameRealTimestampUs) // 上一个frame发生的时间
然后通过frame rate计算出两个frame之间的相距离时间，中间的都透过releaseOneRecordingFrame来drop掉
也就是说driver返回的东西都不变，只是在SW这层我们自己来处理掉
关于Time-lapse相关的可以参阅
https://en.wikipedia.org/wiki/Time-lapse_photography
2. 录影当中需要用到Camera的话是通过ICameraRecordingProxy，即Camera当中的RecordingProxy(这是一个BnCameraRecordingProxy)
当透过binder，将ICameraRecordingProxy传到服务端进程之后，它就变成了Bp，如下：
case SET_CAMERA: {
ALOGV("SET_CAMERA");
CHECK_INTERFACE(IMediaRecorder, data, reply);
sp<ICamera> camera = interface_cast<ICamera>(data.readStrongBinder());
sp<ICameraRecordingProxy> proxy =
interface_cast<ICameraRecordingProxy>(data.readStrongBinder());
reply->writeInt32(setCamera(camera, proxy));
return NO_ERROR;
} break;
在CameraSource当中会这样去使用
// We get the proxy from Camera, not ICamera. We need to get the proxy // to the remote Camera owned by the application. Here mCamera is a // local Camera object created by us. We cannot use the proxy from // mCamera here. mCamera = Camera::create(camera); if (mCamera == 0) return -EBUSY; mCameraRecordingProxy = proxy; mCameraFlags |= FLAGS_HOT_CAMERA;
疑问点：
CameraSource当中这个
List<sp<IMemory> > mFramesBeingEncoded;
有什么用？
每编码完一个frame，CameraSource就会将其保存起来，Buffer被release的时候，会反过来release掉这些frame(s)，这种做法是为了效率么？为什么不编码完一个frame就将其release掉？
另外不得不再感叹下Google经常的delete this;行为，精妙，但是看起来反常！
这是一篇阅读Bitmap相关内容的笔记！
程序运行环境：
Linux KNIGHT 3.0.0-28-generic #45-Ubuntu SMP Wed Nov 14 21:57:26 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
BMP图像文件被分成4个部分：位图文件头(Bitmap File Header)、位图信息头(Bitmap Info Header)、颜色表(Color Map)和位图数据(即图像数据，Data Bits或Data Body)。
这本书(http://vipbase.net/ipbook/chap01.htm)第一章节有讲这些。
biBitCount：每个像素所占的位数(bit)，其值必须为1(黑白图像)、4(16色图)、8(256色)、24(真彩色图)，32(带透明度alpha通道，位于前8位)。
以下这几篇博客有讲些基本的东西。
http://www.cnblogs.com/shengansong/archive/2011/09/23/2186409.html
http://blog.csdn.net/yutianzuijin/article/details/8243343
http://blog.csdn.net/scut1135/article/details/5573395
我有参考以上资料修改出一个可以在x86_64 GNU/Linux编译并运行的程序，注释里面有写些需要注意的地方。
some pitfalls of __attribute__((packed)), plz refer
http://stackoverflow.com/questions/8568432/is-gccs-attribute-packed-pragma-pack-unsafe
http://stackoverflow.com/questions/11770451/what-is-the-meaning-of-attribute-packed-aligned4
http://stackoverflow.com/questions/11667181/why-does-padding-have-to-be-a-power-of-two
http://stackoverflow.com/questions/119123/why-isnt-sizeof-for-a-struct-equal-to-the-sum-of-sizeof-of-each-member
http://en.wikipedia.org/wiki/Data_structure_alignment
Linux下BMP转化为JPEG程序源代码
http://www.linuxidc.com/Linux/2011-03/33193.htm
如果你自己要开发JPEG相关的话，确保这些包是有安装的。
sudo apt-get install libjpeg62 sudo apt-get install libjpeg62-dev
这里记录些经常遇到的有关Android/AOSP/PandaBoard的问题(可能比较小，并且又不知道放哪里的事情)！
1. PandaBoard获取root privilege(adb remount)
/path/to/aosp/out/target/product/panda/root/default.prop
修改设定为ro.secure=0
再次打包镜像，理论上这应该对所有的ROM都管用！
当然我这里是AOSP源码编译然后烧录的，如果只是普通的ROM，想root的话，请参考官方或者网络上的方法！
Stagefright和codec的交互是通过OpenMAX IL来完成的。
OpenMAX IL的规范头文件位于
/path/to/aosp/frameworks/native/include/media/openmax
概述下OpenMAX IL，它包括客户端(Client)，组件(Component)，端口(Port)，隧道化(Tunneled)。
Client是调用Component的对象，Port和Tunneled是Component之间通信的方式，Component之间还可以存在私有的通信方式。
组件(Component)的功能和其定义的端口类型密切相关，通常情况下：只有一个输出端口的，为Source组件；只有一个输入端口的，为Sink组件；有多个输入端口，一个输出端口的为Mux组件；有一个输入端口，多个输出端口的为DeMux组件；输入输出端口各一个组件的为中间处理环节，这是最常见的组件。
在Android 2.2以及之后，Stagefright是作为OpenMAX IL的上层存在的，它跟OpenMAX IL对接，然后OpenMAX IL又封装了具体的codec。
之前我有在http://guoh.org/lifelog/2013/06/android-mediarecorder-architecture/分析过StagefrightRecorder，当时跟codec的集成我没有仔细分析。
这篇文章会详细分析下这个逻辑过程。
首先我们会碰到几个关键的类，第一个就是OMX，并且它的构造方法当中会去创建一个OMXMaster，继续追踪下，发现OMXMaster正如其名字所示，是一个管理者角色。
OMXMaster作为管理者，每添加一个plugin进来，都会记录在
KeyedVector<String8, OMXPluginBase *> mPluginByComponentName; // 重复名字的codec不会再让其加进来，也就是它会去确定该plugin支持多少codec，然后记录下来 List<OMXPluginBase *> mPlugins; // 主要负责释放内存(不与业务相关，只管有添加进来的plugin)
OMXMaster到底会加多少个OMXPlugin进来？
以及到底会有多少个OMXMaster？
有多少个OMX就有多少个OMXMaster，那有多少个OMX呢？
只有一个，参见
sp<IOMX> MediaPlayerService::getOMX() {
Mutex::Autolock autoLock(mLock);
if (mOMX.get() == NULL) {
mOMX = new OMX;
}
return mOMX;
}
那会有多少个OMXPlugin呢？。
一个SoftOMXPlugin，当中可以加载很多codec进来
/path/to/aosp/frameworks/av/media/libstagefright/codecs/
有多少soft codec都在上面这个目录
会有多少个Vendor plugin(即libstagefrighthw)?
答案也是一个，但是它同样也包含有很多个codec(s)进来，以TI的HW为例子：
TIOMXPlugin(即Vendor plugin)在/path/to/aosp/hardware/ti/omap4xxx/libstagefrighthw/这里
但是具体的实现在/path/to/aosp/hardware/ti/omap4xxx/domx/omx_core/(即libOMX_Core.so)当中
假设要实例化一个codec，就需要经过如下步骤：
TIOMXPlugin.cpp(makeComponentInstance) -> OMX_Core_Wrapper.c(TIOMX_GetHandle) -> OMX_Core.c(OMX_GetHandle) -> 动态去load指定的codec的so档案，然后调用它的初始化方法(OMX_ComponentInit)，至于这个方法名，应该是遵循规范得来的(规范上有写这么句话，有这么一个error msg code)。
/** The component specified did not have a "OMX_ComponentInit" or "OMX_ComponentDeInit entry point */ OMX_ErrorInvalidComponent = (OMX_S32) 0x80001004,
其支持的codec(s)在/path/to/aosp/hardware/ti/omap4xxx/domx/omx_proxy_component/
当然codec最终的实现可能还是在其他地方！
以下这些方法都是OpenMAX IL所定义的于OMX Core有关的标准方法，详见OMX_Core.h，TI在OMX_Core.c当中有去实现这些方法，这些可以说是管理OpenMAX IL这一层所封装的Component的接口，当然包括自身的生命周期的管理，以及Component之间的沟通。
OMX_API OMX_ERRORTYPE OMX_Init OMX_API OMX_ERRORTYPE OMX_Deinit OMX_API OMX_ERRORTYPE OMX_ComponentNameEnum OMX_API OMX_ERRORTYPE OMX_GetHandle OMX_API OMX_ERRORTYPE OMX_FreeHandle OMX_API OMX_ERRORTYPE OMX_SetupTunnel OMX_API OMX_ERRORTYPE OMX_GetContentPipe OMX_API OMX_ERRORTYPE OMX_GetComponentsOfRole OMX_API OMX_ERRORTYPE OMX_GetRolesOfComponent
也就是说OMX是操作所有codec的窗口，它透过OMXMaster来管理Soft/Vendor plugin，每个plugin当中有属于自己的codec(s)。
比如典型的，OMX有allocateNode这个方法，这是增加codec时需要的，当然它实质上是去调用OMXMaster -> OMXPlugin -> OMX_Core -> 特定的codec的实例化方法。
status_t OMX::allocateNode(
const char *name, const sp<IOMXObserver> &observer, node_id *node) {
Mutex::Autolock autoLock(mLock);
*node = 0;
OMXNodeInstance *instance = new OMXNodeInstance(this, observer);
OMX_COMPONENTTYPE *handle; // 声明一个OMX_COMPONENTTYPE指针
// 接着makeComponentInstance创建出来，双指针OUT传值
// 最终会发现在codec当中会去调用这样一个方法(对于TI的HW，这个方法位于omx_proxy_common.c当中)
// /*Calling Proxy Common Init() */
// eError = OMX_ProxyCommonInit(hComponent);
// 它会去初始化OMX_COMPONENTTYPE的各个属性，方法，例如OMX_COMPONENTTYPE::SetCallbacks等等
OMX_ERRORTYPE err = mMaster->makeComponentInstance(
name, &OMXNodeInstance::kCallbacks,
instance, &handle);
if (err != OMX_ErrorNone) {
ALOGV("FAILED to allocate omx component '%s'", name);
instance->onGetHandleFailed();
return UNKNOWN_ERROR;
}
*node = makeNodeID(instance);
mDispatchers.add(*node, new CallbackDispatcher(instance));
instance->setHandle(*node, handle);
mLiveNodes.add(observer->asBinder(), instance);
observer->asBinder()->linkToDeath(this);
return OK;
}
有两点值得注意的是
a)
status_t OMXClient::connect() {
sp<IServiceManager> sm = defaultServiceManager();
sp<IBinder> binder = sm->getService(String16("media.player"));
sp<IMediaPlayerService> service = interface_cast<IMediaPlayerService>(binder);
// defaultServiceManager(位于/path/to/aosp/frameworks/native/include/binder/IServiceManager.h)
// 这样调用就跟本地的一个service指针来调用getOMX是一样的，因为本来StatefrightRecorder是在MediaPlayerService当中new出来的
// 这个我们在之前的研究讨论中做过实验，详细参见http://guoh.org/lifelog/2013/06/sample-of-customized-service-on-android-platform/怎样为Android添加一个系统级服务
CHECK(service.get() != NULL);
mOMX = service->getOMX(); // BnOMX
CHECK(mOMX.get() != NULL);
// 什么情况会有不是在本地，貌似OMX是在MediaPlayerService当中new出来的，
// OMXClient是在StagefrightRecorder当中new出来的，而StagefrightRecorder又是在MediaPlayerService当中new出来
// 所以什么情况下会走到如下的这个using client-side OMX mux当中去(MuxOMX位于OMXClient.cpp当中)？
if (!mOMX->livesLocally(NULL /* node */, getpid())) {
ALOGI("Using client-side OMX mux.");
mOMX = new MuxOMX(mOMX); // 继承自IOMX
}
return OK;
}
容易看出OMXClient是获取OMX的窗口
b)
sp<MediaSource> encoder = OMXCodec::Create(
client.interface(), enc_meta,
true /* createEncoder */, cameraSource,
NULL, encoder_flags);
那OMXCodec/
/path/to/aosp/frameworks/av/include/media/stagefright
/path/to/aosp/frameworks/av/media/libstagefright
OMXNodeInstance
/path/to/aosp/frameworks/av/media/libstagefright/include
/path/to/aosp/frameworks/av/media/libstagefright/omx
是做什么的？
OMXCodec，OMXCodecObserver和OMXNodeInstance是一一对应的，
简单的可以理解它们3个构成了OpenMAX IL的一个Component，每一个node就是一个codec在OMX服务端的标识。
当然还有CallbackDispatcher，用于处理codec过来的消息，通过post/dispatch来发起接收，最终透过它的OMXNodeInstance::onMessage -> OMXCodecObserver::onMessage -> OMXCodec::on_message，当然消息的来源是因为我们有向codec注册OMXNodeInstance::kCallbacks，
如下：kCallbacks包含3种事件
OMX_CALLBACKTYPE OMXNodeInstance::kCallbacks = {
&OnEvent, &OnEmptyBufferDone, &OnFillBufferDone
};
它们分别都会调用到自己owner的OnEvent/OnEmptyBufferDone/OnFillBufferDone
// static
OMX_ERRORTYPE OMXNodeInstance::OnEvent(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_EVENTTYPE eEvent,
OMX_IN OMX_U32 nData1,
OMX_IN OMX_U32 nData2,
OMX_IN OMX_PTR pEventData) {
OMXNodeInstance *instance = static_cast<OMXNodeInstance *>(pAppData);
if (instance->mDying) {
return OMX_ErrorNone;
}
return instance->owner()->OnEvent(
instance->nodeID(), eEvent, nData1, nData2, pEventData);
}
而owner相应的又会调用自己dispatcher的post方法，如下：
OMX_ERRORTYPE OMX::OnEvent(
node_id node,
OMX_IN OMX_EVENTTYPE eEvent,
OMX_IN OMX_U32 nData1,
OMX_IN OMX_U32 nData2,
OMX_IN OMX_PTR pEventData) {
ALOGV("OnEvent(%d, %ld, %ld)", eEvent, nData1, nData2);
omx_message msg;
msg.type = omx_message::EVENT;
msg.node = node;
msg.u.event_data.event = eEvent;
msg.u.event_data.data1 = nData1;
msg.u.event_data.data2 = nData2;
findDispatcher(node)->post(msg);
return OMX_ErrorNone;
}
这样所有的事件都串起来了，消息有来源，有最终的去处！
结合这些信息所以我们就可以认为在这里是创建出了一个Component出来。
另外值得提一下的就是我们前面已经讲过Component之间也是可以有信息交互的，比如我们在录影的时候，CameraSource就作为数据的来源被OMXCodec持有，那么从OMXCodec读取的数据实际都是来自于CameraSource，然后就被MediaWriter，例如MPEG4Writer写到文件当中。
OMX_GetComponentVersion OMX_SendCommand OMX_GetParameter OMX_SetParameter OMX_GetConfig OMX_SetConfig OMX_GetExtensionIndex OMX_GetState OMX_UseBuffer OMX_AllocateBuffer OMX_FreeBuffer OMX_EmptyThisBuffer OMX_FillThisBuffer OMX_UseEGLImage
以上这些macro位于OMX_Core.h当中，这是OpenMax所定义的
针对每一个Component封装，他们负责与具体的codec进行沟通，对于TI的HW，这些方法都实现于omx_proxy_common.c当中
其他内容：
1、HardwareAPI.h
extern android::OMXPluginBase *createOMXPlugin();
定义一个实例化plugin的入口方法，在OMXMaster当中回去动态调用这个方法
2、对于Qualcomm的HW来说，它的libstagefrighthw实现位于如下位置：
libomxcore.so /path/to/aosp/hardware/qcom/media/mm-core/omxcore
3、还有比较关键的就是OpenMAX IL之间本身的原理，还有相互沟通是怎么样的，这个比较重要。
都知道OpenMAX IL最大的作用就是一个适配层作用，它封装了底层Vendor容易变化的部分(不同codec/不同Vendor，或者其他形式的变化)，对上层提供一个统一的接口(比如在这里就是Stagefright)。
一个Component的状态分为这么多种，这些定义在OMXCodec当中
enum State {
DEAD,
LOADED,
LOADED_TO_IDLE,
IDLE_TO_EXECUTING,
EXECUTING,
EXECUTING_TO_IDLE,
IDLE_TO_LOADED,
RECONFIGURING,
ERROR
};
Component之间的交互，比如Tunneled communication，是通过Component的ComponentTunnelRequest方法来完成的(TI的HW实现在omx_proxy_common.c当中)，但是从代码上看起来在Qualcomm或者TI的Camera这块都没有用到Tunneled communication，Qualcomm根本就没有去实现ComponentTunnelRequest方法。
Buffer的传输过程，这块初看很复杂，其实流程还是比较清楚的。在Camera录影这个场景中，MediaWriter会不断的请求MediaSource的read方法，从中间读取数据，但是在MediaSource内部会对其进行控制，用的就是signal(Condition mBufferFilled;)。OMXCodec的drainInputBuffer/fillOutputBuffer这两个方法最重要，排空input缓冲(source过来的数据)和填充output缓冲(准备给writer的数据)。drainInputBuffer把数据发到实际的codec，codec收完之后发送EMPTY_BUFFER_DONE消息过来，OMXCodec收到该消息继续发送数据到codec。另外同时还有codec编码完成之后会发送FILL_BUFFER_DONE过来，OMXCodec收到该消息之后就执行fillOutputBuffer，这样MediaWriter就收到编码完成的数据，然后保存就可以了。细节性的东西就需要自己追踪代码可能更好理解，聪明的你可能就想到了那这里是不是存在两个Buffer，是的，你是对的，请看List<size_t> mFilledBuffers;。
以TI的platform为例子，也就是这两个文件
/path/to/aosp/frameworks/av/services/camera/libcameraservice/CameraHardwareInterface.h
和
/path/to/aosp/hardware/ti/omap4xxx/camera/CameraHal.cpp
是如何关联的。
谁来加载camera.$platform$.so，这个是真正的HAL的实现，里面实际会去调用V4L2的东西。
好突然，可能你现在还不知道camera.$platform$.so是什么？是这样的，但是到你看到后面你就会发现Vendor的HAL实现是包成一个so档的，也就是这个。
void CameraService::onFirstRef()
{
BnCameraService::onFirstRef();
if (hw_get_module(CAMERA_HARDWARE_MODULE_ID,
(const hw_module_t **)&mModule) < 0) {
ALOGE("Could not load camera HAL module");
mNumberOfCameras = 0;
}
else {
mNumberOfCameras = mModule->get_number_of_cameras();
if (mNumberOfCameras > MAX_CAMERAS) {
ALOGE("Number of cameras(%d) > MAX_CAMERAS(%d).",
mNumberOfCameras, MAX_CAMERAS);
mNumberOfCameras = MAX_CAMERAS;
}
for (int i = 0; i < mNumberOfCameras; i++) {
setCameraFree(i);
}
}
}
在/path/to/aosp/hardware/libhardware/hardware.c当中有下面这些方法
hw_get_module(const char *id, const struct hw_module_t **module) // 这里定义的是hw_module_t，但是Vendor HAL当中实现的是camera_module_t
/* Loop through the configuration variants looking for a module */
for (i=0 ; i<HAL_VARIANT_KEYS_COUNT+1 ; i++) {
if (i < HAL_VARIANT_KEYS_COUNT) {
if (property_get(variant_keys[i], prop, NULL) == 0) {
continue;
}
snprintf(path, sizeof(path), "%s/%s.%s.so",
HAL_LIBRARY_PATH2, name, prop);
if (access(path, R_OK) == 0) break;
snprintf(path, sizeof(path), "%s/%s.%s.so",
HAL_LIBRARY_PATH1, name, prop);
if (access(path, R_OK) == 0) break;
} else {
snprintf(path, sizeof(path), "%s/%s.default.so",
HAL_LIBRARY_PATH1, name);
if (access(path, R_OK) == 0) break;
}
}
static const char *variant_keys[] = {
"ro.hardware", /* This goes first so that it can pick up a different
file on the emulator. */
"ro.product.board",
"ro.board.platform",
"ro.arch"
};
static const int HAL_VARIANT_KEYS_COUNT =
(sizeof(variant_keys)/sizeof(variant_keys[0]));
/path/to/aosp/out/target/product/panda/system/build.prop
这里面有platform或者arch相关的数据
/*
* load the symbols resolving undefined symbols before
* dlopen returns. Since RTLD_GLOBAL is not or'd in with
* RTLD_NOW the external symbols will not be global
*/
handle = dlopen(path, RTLD_NOW);
/* Get the address of the struct hal_module_info. */
const char *sym = HAL_MODULE_INFO_SYM_AS_STR;
hmi = (struct hw_module_t *)dlsym(handle, sym);
if (hmi == NULL) {
ALOGE("load: couldn't find symbol %s", sym);
status = -EINVAL;
goto done;
}
*pHmi = hmi; // 动态加载Vendor HAL之后要返回这个hw_module_t结构体供Android Service/HAL层使用
这个load出来的HAL_MODULE_INFO_SYM_AS_STR是什么？
先看它是什么，定义位于hardware.h当中
/** * Name of the hal_module_info */ #define HAL_MODULE_INFO_SYM HMI /** * Name of the hal_module_info as a string */ #define HAL_MODULE_INFO_SYM_AS_STR "HMI"
理论上来说dlsym就是会去load一个名字为”HMI”的变量/函数，也就是说在Vendor HAL当中必然有名字为”HMI”的这样一个东西。
看看hw_module_t的定义。
/**
* Every hardware module must have a data structure named HAL_MODULE_INFO_SYM
* and the fields of this data structure must begin with hw_module_t
* followed by module specific information.
*/
typedef struct hw_module_t {
/** tag must be initialized to HARDWARE_MODULE_TAG */
uint32_t tag;
/**
* The API version of the implemented module. The module owner is
* responsible for updating the version when a module interface has
* changed.
*
* The derived modules such as gralloc and audio own and manage this field.
* The module user must interpret the version field to decide whether or
* not to inter-operate with the supplied module implementation.
* For example, SurfaceFlinger is responsible for making sure that
* it knows how to manage different versions of the gralloc-module API,
* and AudioFlinger must know how to do the same for audio-module API.
*
* The module API version should include a major and a minor component.
* For example, version 1.0 could be represented as 0x0100. This format
* implies that versions 0x0100-0x01ff are all API-compatible.
*
* In the future, libhardware will expose a hw_get_module_version()
* (or equivalent) function that will take minimum/maximum supported
* versions as arguments and would be able to reject modules with
* versions outside of the supplied range.
*/
uint16_t module_api_version;
#define version_major module_api_version
/**
* version_major/version_minor defines are supplied here for temporary
* source code compatibility. They will be removed in the next version.
* ALL clients must convert to the new version format.
*/
/**
* The API version of the HAL module interface. This is meant to
* version the hw_module_t, hw_module_methods_t, and hw_device_t
* structures and definitions.
*
* The HAL interface owns this field. Module users/implementations
* must NOT rely on this value for version information.
*
* Presently, 0 is the only valid value.
*/
uint16_t hal_api_version;
#define version_minor hal_api_version
/** Identifier of module */
const char *id;
/** Name of this module */
const char *name;
/** Author/owner/implementor of the module */
const char *author;
/** Modules methods */
struct hw_module_methods_t* methods;
/** module's dso */
void* dso;
/** padding to 128 bytes, reserved for future use */
uint32_t reserved[32-7];
} hw_module_t;
定义位于hardware.h当中
对于TI的hardware来讲，load的就是camera.omap4.so(也就是前面所说的camera.$platform$.so)的名字为HMI的结构体。
这个需要仔细看下才能发现HAL_MODULE_INFO_SYM实际就是HMI，hardware.h当中定义的那个macro。
HMI位于
/path/to/aosp/hardware/ti/omap4xxx/camera/CameraHal_Module.cpp
当中
camera_module_t HAL_MODULE_INFO_SYM = {
common: {
tag: HARDWARE_MODULE_TAG,
version_major: 1,
version_minor: 0,
id: CAMERA_HARDWARE_MODULE_ID,
name: "TI OMAP CameraHal Module",
author: "TI",
methods: &camera_module_methods,
dso: NULL, /* remove compilation warnings */
reserved: {0}, /* remove compilation warnings */
},
get_number_of_cameras: camera_get_number_of_cameras,
get_camera_info: camera_get_camera_info,
};
通过’arm-eabi-objdump’ -t camera.omap4.so | grep ‘HMI’
也可以验证这一点
而再仔细看看这个common实际上就是hw_module_t这个结构体，这也就印证了前面hw_module_t定义的地方所说的
/**
* Every hardware module must have a data structure named HAL_MODULE_INFO_SYM
* and the fields of this data structure must begin with hw_module_t
* followed by module specific information.
*/
这里需要注意的是hardware.h当中定义的都是hw_module_t，但是Vendor HAL实现的都是自己的xxx_module_t，比如方法定义的类型和实际类型不一样，这个能工作有什么样的奥秘？
其实就是指针的第一个地址，Vendor HAL当中定义的结构必须要求第一个域是hw_module_t，这样地址就可以对应起来。
总之Android使用的这种技法需要理解，要不然都无法知道他们之间是怎么串起来的，指针强大，但小心使用，否则也可能伤身！
camera_device_t和hw_device_t也是一样的用法！
#define CAMERA_HARDWARE_MODULE_ID "camera"
定义位于camera_common.h当中
其他一些使用到的宏之类的
#ifndef _LINUX_LIMITS_H #define _LINUX_LIMITS_H #define NR_OPEN 1024 #define NGROUPS_MAX 65536 /* WARNING: DO NOT EDIT, AUTO-GENERATED CODE - SEE TOP FOR INSTRUCTIONS */ #define ARG_MAX 131072 #define CHILD_MAX 999 #define OPEN_MAX 256 #define LINK_MAX 127 /* WARNING: DO NOT EDIT, AUTO-GENERATED CODE - SEE TOP FOR INSTRUCTIONS */ #define MAX_CANON 255 #define MAX_INPUT 255 #define NAME_MAX 255 #define PATH_MAX 4096 /* WARNING: DO NOT EDIT, AUTO-GENERATED CODE - SEE TOP FOR INSTRUCTIONS */ #define PIPE_BUF 4096 #define XATTR_NAME_MAX 255 #define XATTR_SIZE_MAX 65536 #define XATTR_LIST_MAX 65536 /* WARNING: DO NOT EDIT, AUTO-GENERATED CODE - SEE TOP FOR INSTRUCTIONS */ #define RTSIG_MAX 32 #endif
对于Camera HAL来说，你需要找到
typedef struct camera_device {
/**
* camera_device.common.version must be in the range
* HARDWARE_DEVICE_API_VERSION(0,0)-(1,FF). CAMERA_DEVICE_API_VERSION_1_0 is
* recommended.
*/
hw_device_t common;
camera_device_ops_t *ops;
void *priv;
} camera_device_t;
位于camera.h当中
typedef struct camera_module {
hw_module_t common;
int (*get_number_of_cameras)(void);
int (*get_camera_info)(int camera_id, struct camera_info *info);
} camera_module_t;
位于camera_common.h当中
扫平这些障碍之后，后面就会顺利多了。
比如takePicture这个动作，在CameraHal_Module当中会被于CameraHal关联起来，上层调用就会进入到CameraHal当中对应的方法
status_t CameraHal::takePicture( )
{
status_t ret = NO_ERROR;
CameraFrame frame;
CameraAdapter::BuffersDescriptor desc;
int burst;
const char *valstr = NULL;
unsigned int bufferCount = 1;
Mutex::Autolock lock(mLock);
#if PPM_INSTRUMENTATION || PPM_INSTRUMENTATION_ABS
gettimeofday(&mStartCapture, NULL);
#endif
LOG_FUNCTION_NAME;
if(!previewEnabled() && !mDisplayPaused)
{
LOG_FUNCTION_NAME_EXIT;
CAMHAL_LOGEA("Preview not started...");
return NO_INIT;
}
// return error if we are already capturing
if ( (mCameraAdapter->getState() == CameraAdapter::CAPTURE_STATE &&
mCameraAdapter->getNextState() != CameraAdapter::PREVIEW_STATE) ||
(mCameraAdapter->getState() == CameraAdapter::VIDEO_CAPTURE_STATE &&
mCameraAdapter->getNextState() != CameraAdapter::VIDEO_STATE) ) {
CAMHAL_LOGEA("Already capturing an image...");
return NO_INIT;
}
// we only support video snapshot if we are in video mode (recording hint is set)
valstr = mParameters.get(TICameraParameters::KEY_CAP_MODE);
if ( (mCameraAdapter->getState() == CameraAdapter::VIDEO_STATE) &&
(valstr && strcmp(valstr, TICameraParameters::VIDEO_MODE)) ) {
CAMHAL_LOGEA("Trying to capture while recording without recording hint set...");
return INVALID_OPERATION;
}
if ( !mBracketingRunning )
{
if ( NO_ERROR == ret )
{
burst = mParameters.getInt(TICameraParameters::KEY_BURST);
}
//Allocate all buffers only in burst capture case
if ( burst > 1 )
{
bufferCount = CameraHal::NO_BUFFERS_IMAGE_CAPTURE;
if ( NULL != mAppCallbackNotifier.get() )
{
mAppCallbackNotifier->setBurst(true);
}
}
else
{
if ( NULL != mAppCallbackNotifier.get() )
{
mAppCallbackNotifier->setBurst(false);
}
}
// 这就是为什么我们正常拍照，preview就会stop住，拍完一张之后需要去startPreview
// pause preview during normal image capture
// do not pause preview if recording (video state)
if (NO_ERROR == ret &&
NULL != mDisplayAdapter.get() &&
burst < 1) {
if (mCameraAdapter->getState() != CameraAdapter::VIDEO_STATE) {
mDisplayPaused = true;
mPreviewEnabled = false;
ret = mDisplayAdapter->pauseDisplay(mDisplayPaused);
// since preview is paused we should stop sending preview frames too
if(mMsgEnabled & CAMERA_MSG_PREVIEW_FRAME) {
mAppCallbackNotifier->disableMsgType (CAMERA_MSG_PREVIEW_FRAME);
}
}
#if PPM_INSTRUMENTATION || PPM_INSTRUMENTATION_ABS
mDisplayAdapter->setSnapshotTimeRef(&mStartCapture);
#endif
}
// if we taking video snapshot...
if ((NO_ERROR == ret) && (mCameraAdapter->getState() == CameraAdapter::VIDEO_STATE)) {
// enable post view frames if not already enabled so we can internally
// save snapshot frames for generating thumbnail
if((mMsgEnabled & CAMERA_MSG_POSTVIEW_FRAME) == 0) {
mAppCallbackNotifier->enableMsgType(CAMERA_MSG_POSTVIEW_FRAME);
}
}
if ( (NO_ERROR == ret) && (NULL != mCameraAdapter) )
{
if ( NO_ERROR == ret )
ret = mCameraAdapter->sendCommand(CameraAdapter::CAMERA_QUERY_BUFFER_SIZE_IMAGE_CAPTURE,
( int ) &frame,
bufferCount);
if ( NO_ERROR != ret )
{
CAMHAL_LOGEB("CAMERA_QUERY_BUFFER_SIZE_IMAGE_CAPTURE returned error 0x%x", ret);
}
}
if ( NO_ERROR == ret )
{
mParameters.getPictureSize(( int * ) &frame.mWidth,
( int * ) &frame.mHeight);
// 分配buffer
ret = allocImageBufs(frame.mWidth,
frame.mHeight,
frame.mLength,
mParameters.getPictureFormat(),
bufferCount);
if ( NO_ERROR != ret )
{
CAMHAL_LOGEB("allocImageBufs returned error 0x%x", ret);
}
}
if ( (NO_ERROR == ret) && ( NULL != mCameraAdapter ) )
{
desc.mBuffers = mImageBufs;
desc.mOffsets = mImageOffsets;
desc.mFd = mImageFd;
desc.mLength = mImageLength;
desc.mCount = ( size_t ) bufferCount;
desc.mMaxQueueable = ( size_t ) bufferCount;
ret = mCameraAdapter->sendCommand(CameraAdapter::CAMERA_USE_BUFFERS_IMAGE_CAPTURE,
( int ) &desc);
}
}
if ( ( NO_ERROR == ret ) && ( NULL != mCameraAdapter ) )
{
#if PPM_INSTRUMENTATION || PPM_INSTRUMENTATION_ABS
//pass capture timestamp along with the camera adapter command
ret = mCameraAdapter->sendCommand(CameraAdapter::CAMERA_START_IMAGE_CAPTURE, (int) &mStartCapture);
#else
ret = mCameraAdapter->sendCommand(CameraAdapter::CAMERA_START_IMAGE_CAPTURE);
#endif
}
return ret;
}
经过V4LCameraAdapter::UseBuffersPreview
最终会走向V4L2中去
另外CameraAdapter_Factory()在V4LCameraAdapter当中。
在Ice Cream Sandwich当中，Prelinker这个东西就被移除了，理由如下：
commit b375e71d306f2fd356b9b356b636e568c4581fa1
Author: Iliyan Malchev
Date: Tue Mar 8 16:19:48 2011 -0800build: remove prelinker build build system
This patch removes support for prelinking from the build system. By now, the
prelinker has outlived its usefulness for several reasons. Firstly, the
speedup that it afforded in the early days of Android is now nullified by the
speed of hardware, as well as by the presence of Zygote. Secondly, the space
savings that come with prelinking (measued at 17MB on a recent honeycomb
stingray build) are no longer important either. Thirdly, prelinking reduces
the effectiveness of Address-Space-Layout Randomization. Finally, since it is
not part of the gcc suite, the prelinker needs to be maintained separately.The patch deletes apriori, soslim, lsd, isprelinked, and iself from the source
tree. It also removes the prelink map.LOCAL_PRELINK_MODULE becomes a no-op. Individual Android.mk will get cleaned
separately. Support for prelinking will have to be removed from the recovery
code and from the dynamic loader as well.Change-Id: I5839c9c25f7772d5183eedfe20ab924f2a7cd411
有关的参考(为何而生，为何而死)：
https://groups.google.com/forum/#!topic/android-building/UGJvYMZaVqw
http://en.wikipedia.org/wiki/Address_space_layout_randomization
http://people.redhat.com/jakub/prelink/prelink.pdf
http://jserv.blogspot.com/2010/10/android.html
所以现在还在讨论这个的用处的应该可以收手了。
想要知道Driver回来的数据是如何显示出来的吗？
这个工作是在哪一层做的？
跟踪代码就可以理清楚这一过程，这里代码版本是Jelly Bean。
CameraHardwareInterface初始化的时候会初始化好window有关的数据
struct camera_preview_window {
struct preview_stream_ops nw;
void *user;
};
struct camera_preview_window mHalPreviewWindow;
status_t initialize(hw_module_t *module)
{
ALOGI("Opening camera %s", mName.string());
int rc = module->methods->open(module, mName.string(),
(hw_device_t **)&mDevice);
if (rc != OK) {
ALOGE("Could not open camera %s: %d", mName.string(), rc);
return rc;
}
initHalPreviewWindow();
return rc;
}
void initHalPreviewWindow()
{
mHalPreviewWindow.nw.cancel_buffer = __cancel_buffer;
mHalPreviewWindow.nw.lock_buffer = __lock_buffer;
mHalPreviewWindow.nw.dequeue_buffer = __dequeue_buffer;
mHalPreviewWindow.nw.enqueue_buffer = __enqueue_buffer;
mHalPreviewWindow.nw.set_buffer_count = __set_buffer_count;
mHalPreviewWindow.nw.set_buffers_geometry = __set_buffers_geometry;
mHalPreviewWindow.nw.set_crop = __set_crop;
mHalPreviewWindow.nw.set_timestamp = __set_timestamp;
mHalPreviewWindow.nw.set_usage = __set_usage;
mHalPreviewWindow.nw.set_swap_interval = __set_swap_interval;
mHalPreviewWindow.nw.get_min_undequeued_buffer_count =
__get_min_undequeued_buffer_count;
}
static int __dequeue_buffer(struct preview_stream_ops* w,
buffer_handle_t** buffer, int *stride)
{
int rc;
ANativeWindow *a = anw(w);
ANativeWindowBuffer* anb;
rc = native_window_dequeue_buffer_and_wait(a, &anb);
if (!rc) {
*buffer = &anb->handle;
*stride = anb->stride;
}
return rc;
}
static int __enqueue_buffer(struct preview_stream_ops* w,
buffer_handle_t* buffer)
{
ANativeWindow *a = anw(w);
return a->queueBuffer(a,
container_of(buffer, ANativeWindowBuffer, handle), -1);
}
继而在Vendor的HAL当中，这里以TI为例子
/**
@brief Sets ANativeWindow object.
Preview buffers provided to CameraHal via this object. DisplayAdapter will be interfacing with it
to render buffers to display.
@param[in] window The ANativeWindow object created by Surface flinger
@return NO_ERROR If the ANativeWindow object passes validation criteria
@todo Define validation criteria for ANativeWindow object. Define error codes for scenarios
*/
status_t CameraHal::setPreviewWindow(struct preview_stream_ops *window)
{
status_t ret = NO_ERROR;
CameraAdapter::BuffersDescriptor desc;
LOG_FUNCTION_NAME;
mSetPreviewWindowCalled = true;
///If the Camera service passes a null window, we destroy existing window and free the DisplayAdapter
if(!window)
{
if(mDisplayAdapter.get() != NULL)
{
///NULL window passed, destroy the display adapter if present
CAMHAL_LOGDA("NULL window passed, destroying display adapter");
mDisplayAdapter.clear();
///@remarks If there was a window previously existing, we usually expect another valid window to be passed by the client
///@remarks so, we will wait until it passes a valid window to begin the preview again
mSetPreviewWindowCalled = false;
}
CAMHAL_LOGDA("NULL ANativeWindow passed to setPreviewWindow");
return NO_ERROR;
}else if(mDisplayAdapter.get() == NULL)
{
// Need to create the display adapter since it has not been created
// Create display adapter
mDisplayAdapter = new ANativeWindowDisplayAdapter();
...
// DisplayAdapter needs to know where to get the CameraFrames from inorder to display
// Since CameraAdapter is the one that provides the frames, set it as the frame provider for DisplayAdapter
mDisplayAdapter->setFrameProvider(mCameraAdapter);
...
// Update the display adapter with the new window that is passed from CameraService
// 这里的window(ANativeWindowDisplayAdapter::mANativeWindow)就是CameraHardwareInterface::mHalPreviewWindow::nw
ret = mDisplayAdapter->setPreviewWindow(window);
if(ret!=NO_ERROR)
{
CAMHAL_LOGEB("DisplayAdapter setPreviewWindow returned error %d", ret);
}
if(mPreviewStartInProgress)
{
CAMHAL_LOGDA("setPreviewWindow called when preview running");
// Start the preview since the window is now available
ret = startPreview();
}
} else {
// Update the display adapter with the new window that is passed from CameraService
ret = mDisplayAdapter->setPreviewWindow(window);
if ( (NO_ERROR == ret) && previewEnabled() ) {
restartPreview();
} else if (ret == ALREADY_EXISTS) {
// ALREADY_EXISTS should be treated as a noop in this case
ret = NO_ERROR;
}
}
LOG_FUNCTION_NAME_EXIT;
return ret;
}
注意看setPreviewWindow之前我加的一行中文注释
在TI的HAL当中有去做包一些adaper出来，实际就是取数据的(比如V4L2CameraAdapter，也就是CameraAdapter)，和数据发其他地方(比如ANativeWindowDisplayAdapter，也就是DisplayAdapter)去用的。
在TI的HW当中，有previewThread(位于V4LCameraAdapter当中)，previewThread会不断的通过IOCTL从V4L2获取frame
int V4LCameraAdapter::previewThread()
{
status_t ret = NO_ERROR;
int width, height;
CameraFrame frame;
if (mPreviewing)
{
int index = 0;
char *fp = this->GetFrame(index);
uint8_t* ptr = (uint8_t*) mPreviewBufs.keyAt(index);
int width, height;
uint16_t* dest = (uint16_t*)ptr;
uint16_t* src = (uint16_t*) fp;
mParams.getPreviewSize(&width, &height);
...
mParams.getPreviewSize(&width, &height);
frame.mFrameType = CameraFrame::PREVIEW_FRAME_SYNC;
frame.mBuffer = ptr;
...
ret = sendFrameToSubscribers(&frame);
}
return ret;
}
char * V4LCameraAdapter::GetFrame(int &index)
{
int ret;
mVideoInfo->buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
mVideoInfo->buf.memory = V4L2_MEMORY_MMAP;
/* DQ */
ret = ioctl(mCameraHandle, VIDIOC_DQBUF, &mVideoInfo->buf);
if (ret < 0) {
CAMHAL_LOGEA("GetFrame: VIDIOC_DQBUF Failed");
return NULL;
}
nDequeued++;
index = mVideoInfo->buf.index;
return (char *)mVideoInfo->mem[mVideoInfo->buf.index];
}
class CameraFrame
{
public:
enum FrameType
{
PREVIEW_FRAME_SYNC = 0x1, ///SYNC implies that the frame needs to be explicitly returned after consuming in order to be filled by camera again
PREVIEW_FRAME = 0x2 , ///Preview frame includes viewfinder and snapshot frames
IMAGE_FRAME_SYNC = 0x4, ///Image Frame is the image capture output frame
IMAGE_FRAME = 0x8,
VIDEO_FRAME_SYNC = 0x10, ///Timestamp will be updated for these frames
VIDEO_FRAME = 0x20,
FRAME_DATA_SYNC = 0x40, ///Any extra data assosicated with the frame. Always synced with the frame
FRAME_DATA= 0x80,
RAW_FRAME = 0x100,
SNAPSHOT_FRAME = 0x200,
ALL_FRAMES = 0xFFFF ///Maximum of 16 frame types supported
};
enum FrameQuirks
{
ENCODE_RAW_YUV422I_TO_JPEG = 0x1 << 0,
HAS_EXIF_DATA = 0x1 << 1,
};
//default contrustor
CameraFrame():
...
{
...
}
//copy constructor
CameraFrame(const CameraFrame &frame) :
...
{
...
}
void *mCookie;
void *mCookie2;
void *mBuffer;
int mFrameType;
nsecs_t mTimestamp;
unsigned int mWidth, mHeight;
uint32_t mOffset;
unsigned int mAlignment;
int mFd;
size_t mLength;
unsigned mFrameMask;
unsigned int mQuirks;
unsigned int mYuv[2];
///@todo add other member vars like stride etc
};
对于ANativeWindowDisplayAdapter，它有给自己配置一个FrameProvider，实际就是会去注册一种callback，当从driver回来数据的时候会去通知对应的callback，当然这个时候还没有真正的去注册
int ANativeWindowDisplayAdapter::setFrameProvider(FrameNotifier *frameProvider)
{
LOG_FUNCTION_NAME;
// Check for NULL pointer
if ( !frameProvider ) {
CAMHAL_LOGEA("NULL passed for frame provider");
LOG_FUNCTION_NAME_EXIT;
return BAD_VALUE;
}
//Release any previous frame providers
if ( NULL != mFrameProvider ) {
delete mFrameProvider;
}
/** Dont do anything here, Just save the pointer for use when display is
actually enabled or disabled
*/
mFrameProvider = new FrameProvider(frameProvider, this, frameCallbackRelay);
LOG_FUNCTION_NAME_EXIT;
return NO_ERROR;
}
并且我们在开始start preview的时候会去注册这个callback，当然同时在start preview的时候我们也会去分配一块buffer，这样driver回来的数据才有地方放，申请buffer如下
CameraHal::allocateBuffer(…) __dequeue_buffer
下面是启动living preview，注册callback
int ANativeWindowDisplayAdapter::enableDisplay(int width, int height, struct timeval *refTime, S3DParameters *s3dParams)
{
Semaphore sem;
TIUTILS::Message msg;
LOG_FUNCTION_NAME;
if ( mDisplayEnabled )
{
CAMHAL_LOGDA("Display is already enabled");
LOG_FUNCTION_NAME_EXIT;
return NO_ERROR;
}
#if 0 //TODO: s3d is not part of bringup...will reenable
if (s3dParams)
mOverlay->set_s3d_params(s3dParams->mode, s3dParams->framePacking,
s3dParams->order, s3dParams->subSampling);
#endif
#if PPM_INSTRUMENTATION || PPM_INSTRUMENTATION_ABS
if ( NULL != refTime )
{
Mutex::Autolock lock(mLock);
memcpy(&mStandbyToShot, refTime, sizeof(struct timeval));
mMeasureStandby = true;
}
#endif
//Send START_DISPLAY COMMAND to display thread. Display thread will start and then wait for a message
sem.Create();
msg.command = DisplayThread::DISPLAY_START;
// Send the semaphore to signal once the command is completed
msg.arg1 = &sem;
///Post the message to display thread
mDisplayThread->msgQ().put(&msg);
///Wait for the ACK - implies that the thread is now started and waiting for frames
sem.Wait();
// Register with the frame provider for frames
mFrameProvider->enableFrameNotification(CameraFrame::PREVIEW_FRAME_SYNC);
mDisplayEnabled = true;
mPreviewWidth = width;
mPreviewHeight = height;
CAMHAL_LOGVB("mPreviewWidth = %d mPreviewHeight = %d", mPreviewWidth, mPreviewHeight);
LOG_FUNCTION_NAME_EXIT;
return NO_ERROR;
}
int FrameProvider::enableFrameNotification(int32_t frameTypes)
{
LOG_FUNCTION_NAME;
status_t ret = NO_ERROR;
///Enable the frame notification to CameraAdapter (which implements FrameNotifier interface)
mFrameNotifier->enableMsgType(frameTypes<<MessageNotifier::FRAME_BIT_FIELD_POSITION
, mFrameCallback
, NULL
, mCookie
);
// 把frameCallbackRelay这个callback添加到BaseCameraAdapter::mFrameSubscribers当中
// 详见 preview_stream_ops_t* mANativeWindow; 以及 ANativeWindowDisplayAdapter::PostFrame(...)
// __enqueue_buffer
// 最终会把数据更新到CameraHardwareInterface的sp<ANativeWindow> mPreviewWindow;
// 这样每过来一个frame，也就是去更新我们看到的preview数据
LOG_FUNCTION_NAME_EXIT;
return ret;
}
见上面代码中嵌的中文注释，这是我加的。
另外displayThread(位于ANativeWindowDisplayAdapter当中)，displayThread会听一些从Camera HAL来的消息，然后执行，这个不是很复杂。
也就是说数据是怎么贴到view上去，这是HAL这层做掉的，虽然只是一个调用ANativeWindow的过程！
这只是一个粗糙的过程，再深层细节的我们暂时不研究。
举个例子来看下，codec是如何和OpenMAX IL沟通的，这其实应该是Stagefright和codec的交互这篇博客当中的，想想比较独立，又比较长，干脆单独拿出来写一下。
通常来说一个codec(encoder或者decoder)无论怎么复杂，它总会提供一个接口来是我们输入一段数据，并且提供一个接口我们来获取一段数据，因为它的功能就是这样的。
所以我们要把一个codec使用OpenMAX IL的方式集成进来，对我们来说工作量不算很大。这里以FLAC encoder来举个例子(因为它比较简单)，FLAC本身这个codec相关的信息请参见http://xiph.org/flac/，在Android当中源码位于/path/to/aosp/external/flac/，在Android当中OpenMAX IL与它交互是通过libstagefright_soft_flacenc.so这个东西，我们今天就重点关注这个，它的源码位于/path/to/aosp/frameworks/av/media/libstagefright/codecs/flac/enc/。
对Android当中Stagefright和OpenMAX IL不熟悉的话，还是建议先看文章开始提到的那篇。
首先SoftFlacEncoder继承自SimpleSoftOMXComponent，重写了4个方法，分别是
virtual OMX_ERRORTYPE initCheck() const; // 在SoftOMXComponent当中是空实现，自己的codec需要实现来让他人通过这个方法来探测你的codec是否正常
virtual OMX_ERRORTYPE internalGetParameter(
OMX_INDEXTYPE index, OMX_PTR params); // 就是OpenMax IL组件的getParameter
virtual OMX_ERRORTYPE internalSetParameter(
OMX_INDEXTYPE index, const OMX_PTR params); // 就是OpenMax IL组件的setParameter
virtual void onQueueFilled(OMX_U32 portIndex); // 就是OpenMax IL组件的emptyThisBuffer和fillThisBuffer，如果你不清楚，并且还在看这篇文章的话，那你一定是在当散文看 ^_^
这四个方法如果你不明白，就先翻翻OpenMax IL/Stagefright相关代码或者规范。
另外它还有几个私有方法
void initPorts(); // OpenMAX IL通信需要port
OMX_ERRORTYPE configureEncoder(); // FLAC本身也需要些配置，比如把下面的callback设置进去
// FLAC encoder callbacks
// maps to encoderEncodeFlac()
static FLAC__StreamEncoderWriteStatus flacEncoderWriteCallback(
const FLAC__StreamEncoder *encoder, const FLAC__byte buffer[],
size_t bytes, unsigned samples, unsigned current_frame, void *client_data); // 这个是实际丢下去的callback
FLAC__StreamEncoderWriteStatus onEncodedFlacAvailable(
const FLAC__byte buffer[],
size_t bytes, unsigned samples, unsigned current_frame); // 这个只是做了个C向C++的转换
上面这两个方法是callback函数，其实就一个callback，因为如下
// static
FLAC__StreamEncoderWriteStatus SoftFlacEncoder::flacEncoderWriteCallback(
const FLAC__StreamEncoder *encoder, const FLAC__byte buffer[],
size_t bytes, unsigned samples, unsigned current_frame, void *client_data) {
return ((SoftFlacEncoder*) client_data)->onEncodedFlacAvailable(
buffer, bytes, samples, current_frame);
}
为什么要有callback？之前有讲过一个codec总有一个输入和输出，在这里FLAC有给我们一个函数FLAC__stream_encoder_process_interleaved(…)来输入数据，提供了一个callback来输出数据，仅仅这样！
可以参照下FLAC头文件的说明
/** Submit data for encoding. * This version allows you to supply the input data where the channels * are interleaved into a single array (i.e. channel0_sample0, * channel1_sample0, ... , channelN_sample0, channel0_sample1, ...). * The samples need not be block-aligned but they must be * sample-aligned, i.e. the first value should be channel0_sample0 * and the last value channelN_sampleM. Each sample should be a signed * integer, right-justified to the resolution set by * FLAC__stream_encoder_set_bits_per_sample(). For example, if the * resolution is 16 bits per sample, the samples should all be in the * range [-32768,32767]. * * For applications where channel order is important, channels must * follow the order as described in the * <A HREF="../format.html#frame_header">frame header</A>. * * \param encoder An initialized encoder instance in the OK state. * \param buffer An array of channel-interleaved data (see above). * \param samples The number of samples in one channel, the same as for * FLAC__stream_encoder_process(). For example, if * encoding two channels, \c 1000 \a samples corresponds * to a \a buffer of 2000 values. * \assert * \code encoder != NULL \endcode * \code FLAC__stream_encoder_get_state(encoder) == FLAC__STREAM_ENCODER_OK \endcode * \retval FLAC__bool * \c true if successful, else \c false; in this case, check the * encoder state with FLAC__stream_encoder_get_state() to see what * went wrong. */ FLAC_API FLAC__bool FLAC__stream_encoder_process_interleaved(FLAC__StreamEncoder *encoder, const FLAC__int32 buffer[], unsigned samples);
这个就是注册callback的地方，我们注册的是FLAC__StreamEncoderWriteCallback这个callback。
/** Initialize the encoder instance to encode native FLAC streams. * * This flavor of initialization sets up the encoder to encode to a * native FLAC stream. I/O is performed via callbacks to the client. * For encoding to a plain file via filename or open \c FILE*, * FLAC__stream_encoder_init_file() and FLAC__stream_encoder_init_FILE() * provide a simpler interface. * * This function should be called after FLAC__stream_encoder_new() and * FLAC__stream_encoder_set_*() but before FLAC__stream_encoder_process() * or FLAC__stream_encoder_process_interleaved(). * initialization succeeded. * * The call to FLAC__stream_encoder_init_stream() currently will also * immediately call the write callback several times, once with the \c fLaC * signature, and once for each encoded metadata block. * * \param encoder An uninitialized encoder instance. * \param write_callback See FLAC__StreamEncoderWriteCallback. This * pointer must not be \c NULL. * \param seek_callback See FLAC__StreamEncoderSeekCallback. This * pointer may be \c NULL if seeking is not * supported. The encoder uses seeking to go back * and write some some stream statistics to the * STREAMINFO block; this is recommended but not * necessary to create a valid FLAC stream. If * \a seek_callback is not \c NULL then a * \a tell_callback must also be supplied. * Alternatively, a dummy seek callback that just * returns \c FLAC__STREAM_ENCODER_SEEK_STATUS_UNSUPPORTED * may also be supplied, all though this is slightly * less efficient for the encoder. * \param tell_callback See FLAC__StreamEncoderTellCallback. This * pointer may be \c NULL if seeking is not * supported. If \a seek_callback is \c NULL then * this argument will be ignored. If * \a seek_callback is not \c NULL then a * \a tell_callback must also be supplied. * Alternatively, a dummy tell callback that just * returns \c FLAC__STREAM_ENCODER_TELL_STATUS_UNSUPPORTED * may also be supplied, all though this is slightly * less efficient for the encoder. * \param metadata_callback See FLAC__StreamEncoderMetadataCallback. This * pointer may be \c NULL if the callback is not * desired. If the client provides a seek callback, * this function is not necessary as the encoder * will automatically seek back and update the * STREAMINFO block. It may also be \c NULL if the * client does not support seeking, since it will * have no way of going back to update the * STREAMINFO. However the client can still supply * a callback if it would like to know the details * from the STREAMINFO. * \param client_data This value will be supplied to callbacks in their * \a client_data argument. * \assert * \code encoder != NULL \endcode * \retval FLAC__StreamEncoderInitStatus * \c FLAC__STREAM_ENCODER_INIT_STATUS_OK if initialization was successful; * see FLAC__StreamEncoderInitStatus for the meanings of other return values. */ FLAC_API FLAC__StreamEncoderInitStatus FLAC__stream_encoder_init_stream(FLAC__StreamEncoder *encoder, FLAC__StreamEncoderWriteCallback write_callback, FLAC__StreamEncoderSeekCallback seek_callback, FLAC__StreamEncoderTellCallback tell_callback, FLAC__StreamEncoderMetadataCallback metadata_callback, void *client_data);
简单介绍下这代码的意思
initPorts()初始化两个ports，0为输入，1为输出，记住一点很重要，谁申请的内存，谁释放，这点对于理解层次复杂的这套系统来说很重要。
最重要的是下面这个方法，上层把buffer满之后会call到这里，上次需要获取处理好的buffer也会call到这里
void SoftFlacEncoder::onQueueFilled(OMX_U32 portIndex) {
ALOGV("SoftFlacEncoder::onQueueFilled(portIndex=%ld)", portIndex);
if (mSignalledError) { // 出错了
return;
}
List<BufferInfo *> &inQueue = getPortQueue(0); // 输入口，好奇这里一次会不会有多个buffer，理论上来同一次就一个
List<BufferInfo *> &outQueue = getPortQueue(1); // 输出的
while (!inQueue.empty() && !outQueue.empty()) {
BufferInfo *inInfo = *inQueue.begin();
OMX_BUFFERHEADERTYPE *inHeader = inInfo->mHeader;
BufferInfo *outInfo = *outQueue.begin();
OMX_BUFFERHEADERTYPE *outHeader = outInfo->mHeader;
if (inHeader->nFlags & OMX_BUFFERFLAG_EOS) { // 编解码结束的时候，都清空标记，notifyEmptyBufferDone和notifyFillBufferDone都返回
inQueue.erase(inQueue.begin()); // 这是我们自己申请的内存，自己释放
inInfo->mOwnedByUs = false;
notifyEmptyBufferDone(inHeader);
outHeader->nFilledLen = 0;
outHeader->nFlags = OMX_BUFFERFLAG_EOS;
outQueue.erase(outQueue.begin()); // 这是我们自己申请的内存，自己释放
outInfo->mOwnedByUs = false;
notifyFillBufferDone(outHeader);
return;
}
if (inHeader->nFilledLen > kMaxNumSamplesPerFrame * sizeof(FLAC__int32) * 2) {
ALOGE("input buffer too large (%ld).", inHeader->nFilledLen);
mSignalledError = true;
notify(OMX_EventError, OMX_ErrorUndefined, 0, NULL);
return;
}
assert(mNumChannels != 0);
mEncoderWriteData = true;
mEncoderReturnedEncodedData = false;
mEncoderReturnedNbBytes = 0;
mCurrentInputTimeStamp = inHeader->nTimeStamp;
const unsigned nbInputFrames = inHeader->nFilledLen / (2 * mNumChannels);
const unsigned nbInputSamples = inHeader->nFilledLen / 2;
const OMX_S16 * const pcm16 = reinterpret_cast<OMX_S16 *>(inHeader->pBuffer); // OMX_BUFFERHEADERTYPE当中pBuffer默认为8位，这里强制按16为重新分组，所以一个buffer当中samples的数量就是nFilledLen/2
// 可是为什么要按照16位重新分组？因为设置给codec的FLAC__stream_encoder_set_bits_per_sample(mFlacStreamEncoder, 16);
for (unsigned i=0 ; i < nbInputSamples ; i++) {
mInputBufferPcm32[i] = (FLAC__int32) pcm16[i]; // 按32位解析，因为FLAC需要
}
ALOGV(" about to encode %u samples per channel", nbInputFrames);
FLAC__bool ok = FLAC__stream_encoder_process_interleaved(
mFlacStreamEncoder,
mInputBufferPcm32,
nbInputFrames /*samples per channel*/ ); // 在这里编码，因为FLAC对齐的关系，对输入的buffer需要做一定的转换，也就是上面标记出来的两处
if (ok) {
// 之前注册的callback应该是block的，也就是在FLAC__stream_encoder_process_interleaved返回callback因该也返回了
if (mEncoderReturnedEncodedData && (mEncoderReturnedNbBytes != 0)) {
ALOGV(" dequeueing buffer on output port after writing data");
outInfo->mOwnedByUs = false;
outQueue.erase(outQueue.begin());
outInfo = NULL;
notifyFillBufferDone(outHeader); // 编码好的数据通过这里就回到了上层
outHeader = NULL;
mEncoderReturnedEncodedData = false;
} else {
ALOGV(" encoder process_interleaved returned without data to write");
}
} else {
ALOGE(" error encountered during encoding");
mSignalledError = true;
notify(OMX_EventError, OMX_ErrorUndefined, 0, NULL);
return;
}
inInfo->mOwnedByUs = false;
inQueue.erase(inQueue.begin());
inInfo = NULL;
notifyEmptyBufferDone(inHeader); // 告诉上层，FLAC codec已经取走了in buffer里面的数据，这样看来这里就是上层送数据，codec编码，codec送数据，然后开始第二轮
// 不知道有没有那种实现，就是上层送数据，只要codec告诉上层已经取走了数据，上层也可以继续送数据，即使codec这个时候还没有编码完毕，还没有把上一次的数据传送给上层，也许这样在某些情况下可以加快互等拷贝/准备数据的时间，不过这是soft编码，都会占用CPU的时间，所以多CPU上也许可以改进，但那样应该会复杂些
inHeader = NULL;
}
}
接下来看callback里面
FLAC__StreamEncoderWriteStatus SoftFlacEncoder::onEncodedFlacAvailable(
const FLAC__byte buffer[],
size_t bytes, unsigned samples, unsigned current_frame) {
ALOGV("SoftFlacEncoder::onEncodedFlacAvailable(bytes=%d, samples=%d, curr_frame=%d)",
bytes, samples, current_frame);
#ifdef WRITE_FLAC_HEADER_IN_FIRST_BUFFER
if (samples == 0) {
ALOGI(" saving %d bytes of header", bytes);
memcpy(mHeader + mHeaderOffset, buffer, bytes);
mHeaderOffset += bytes;// will contain header size when finished receiving header
return FLAC__STREAM_ENCODER_WRITE_STATUS_OK;
}
#endif
if ((samples == 0) || !mEncoderWriteData) {
// called by the encoder because there's header data to save, but it's not the role
// of this component (unless WRITE_FLAC_HEADER_IN_FIRST_BUFFER is defined)
ALOGV("ignoring %d bytes of header data (samples=%d)", bytes, samples);
return FLAC__STREAM_ENCODER_WRITE_STATUS_OK;
}
List<BufferInfo *> &outQueue = getPortQueue(1);
CHECK(!outQueue.empty());
BufferInfo *outInfo = *outQueue.begin();
OMX_BUFFERHEADERTYPE *outHeader = outInfo->mHeader;
#ifdef WRITE_FLAC_HEADER_IN_FIRST_BUFFER
if (!mWroteHeader) {
ALOGI(" writing %d bytes of header on output port", mHeaderOffset);
memcpy(outHeader->pBuffer + outHeader->nOffset + outHeader->nFilledLen,
mHeader, mHeaderOffset);
outHeader->nFilledLen += mHeaderOffset;
outHeader->nOffset += mHeaderOffset;
mWroteHeader = true;
}
#endif
// write encoded data
ALOGV(" writing %d bytes of encoded data on output port", bytes);
if (bytes > outHeader->nAllocLen - outHeader->nOffset - outHeader->nFilledLen) {
ALOGE(" not enough space left to write encoded data, dropping %u bytes", bytes);
// a fatal error would stop the encoding
return FLAC__STREAM_ENCODER_WRITE_STATUS_OK;
}
memcpy(outHeader->pBuffer + outHeader->nOffset, buffer, bytes); // 拷贝数据到OpenMAX IL的buffer里面
outHeader->nTimeStamp = mCurrentInputTimeStamp;
outHeader->nOffset = 0;
outHeader->nFilledLen += bytes;
outHeader->nFlags = 0;
mEncoderReturnedEncodedData = true;
mEncoderReturnedNbBytes += bytes;
return FLAC__STREAM_ENCODER_WRITE_STATUS_OK;
}
需要注意的地方
// FLAC takes samples aligned on 32bit boundaries, use this buffer for the conversion // before passing the input data to the encoder FLAC__int32* mInputBufferPcm32; // 这个问题上面已经分析过了
短短几百行代码，这个算简单！
Android在4.2的时候对Camera HAL做了比较大的改动，基本是废弃了原先的CameraHardwareInterface，又弄了一套新的。所以它提供了两种方式实现，根据厂商实现HAL的版本在Camera Service层自动加载对应版本的fwk HAL。目前这块的介绍还是比较少，实现的厂商也比较少，大概有Qualcomm和Samsung在其platform上有去实现。
以下文字是以Qualcomm平台为基础，在AOSP代码基础上得出的一些理解，存在错误的地方请指出，本文也会随着进一步的学习而对错误的地方进行修正！
我有观看这样一个介绍的视频(题外话，开挂国家的男人讲的英语基本听不懂)，自己想法出去看，因为视频无法下载下来，所以我截图了一些，放在Flickr上。
www.youtube.com/watch?v=Lald5txnoHw
以下是一段有关这段视频以及HAL 2.0的简单介绍
The Linux Foundation
Android Builders Summit 2013
Camera 2.0: The New Camera Hardware Interface in Android 4.2
By Balwinder Kaur & Ashutosh Gupta
San Francisco, CaliforniaAndroid 4.2 was released with a new Camera Hardware Abstraction Layer (HAL) Camera 2.0. Camera 2.0 has a big emphasis on collection and providing metadata associated with each frame. It also provides the ability to re-process streams. Although the APIs at the SDK level are yet to expose any new APIS to the end user, the Camera HAL and the Camera Service architecture has been revamped to a different architecture. This presentation provides an insight into the new architecture as well as covering some of the challenges faced in building production quality Camera HAL implementations.
The intended audience for this conference session is engineers who want to learn more about the Android Camera Internals. This talk should facilitate engineers wanting to integrate, improve or innovate using the Camera subsystem.
以下是我自己的一些理解。
HAL2如何同DRV沟通的？
HAL2 fwk和Vendor implementation是通过/path/to/aosp/hardware/qcom/camera/QCamera/HAL2/wrapper/QualcommCamera.cpp来绑定的，初始化名字为HMI的struct，这个跟原先的是一样的。
static hw_module_methods_t camera_module_methods = {
open: camera_device_open,
};
static hw_module_t camera_common = {
tag: HARDWARE_MODULE_TAG,
module_api_version: CAMERA_MODULE_API_VERSION_2_0,
hal_api_version: HARDWARE_HAL_API_VERSION,
id: CAMERA_HARDWARE_MODULE_ID,
name: "Qcamera",
author:"Qcom",
methods: &camera_module_methods,
dso: NULL,
reserved: {0},
};
camera_module_t HAL_MODULE_INFO_SYM = {
common: camera_common,
get_number_of_cameras: get_number_of_cameras,
get_camera_info: get_camera_info,
};
camera2_device_ops_t camera_ops = {
set_request_queue_src_ops: android::set_request_queue_src_ops,
notify_request_queue_not_empty: android::notify_request_queue_not_empty,
set_frame_queue_dst_ops: android::set_frame_queue_dst_ops,
get_in_progress_count: android::get_in_progress_count,
flush_captures_in_progress: android::flush_captures_in_progress,
construct_default_request: android::construct_default_request,
allocate_stream: android::allocate_stream,
register_stream_buffers: android::register_stream_buffers,
release_stream: android::release_stream,
allocate_reprocess_stream: android::allocate_reprocess_stream,
allocate_reprocess_stream_from_stream: android::allocate_reprocess_stream_from_stream,
release_reprocess_stream: android::release_reprocess_stream,
trigger_action: android::trigger_action,
set_notify_callback: android::set_notify_callback,
get_metadata_vendor_tag_ops: android::get_metadata_vendor_tag_ops,
dump: android::dump,
};
QCameraHWI当中
QCameraHardwareInterface::
QCameraHardwareInterface(int cameraId, int mode)
: mCameraId(cameraId)
{
cam_ctrl_dimension_t mDimension;
/* Open camera stack! */
memset(&mMemHooks, 0, sizeof(mm_camear_mem_vtbl_t));
mMemHooks.user_data=this;
mMemHooks.get_buf=get_buffer_hook;
mMemHooks.put_buf=put_buffer_hook;
mCameraHandle=camera_open(mCameraId, &mMemHooks);
ALOGV("Cam open returned %p",mCameraHandle);
if(mCameraHandle == NULL) {
ALOGE("startCamera: cam_ops_open failed: id = %d", mCameraId);
return;
}
mCameraHandle->ops->sync(mCameraHandle->camera_handle);
mChannelId=mCameraHandle->ops->ch_acquire(mCameraHandle->camera_handle);
if(mChannelId<=0)
{
ALOGE("%s:Channel aquire failed",__func__);
mCameraHandle->ops->camera_close(mCameraHandle->camera_handle);
return;
}
/* Initialize # of frame requests the HAL is handling to zero*/
mPendingRequests=0;
}
调用mm_camera_interface当中的camera_open()
进而调用mm_camera当中的mm_camera_open()
这里还是是通过V4L2去同DRV沟通的
但是为什么wrapper当中有一些操作Qualcomm没有去实现呢？
int trigger_action(const struct camera2_device *,
uint32_t trigger_id,
int32_t ext1,
int32_t ext2)
{
return INVALID_OPERATION;
}
比如这个上面理论上来说auto focus等等一些操作需要通过它来触发，但是它却是stub实现，这些不是必须的，还是藏在了某个角落我没有发现？
目前用的是libmmcamera_interface还是libmmcamera_interface2？
从代码看应该是libmmcamera_interface
再看一个实例，start preview 这是一个怎样的过程？
Camera2Client::startPreview(…)
Camera2Client::startPreviewL(…)
StreamingProcessor::updatePreviewStream(…)
Camera2Device::createStream(…)
Camera2Device::StreamAdapter::connectToDevice(…)
camera2_device_t->ops->allocate_stream(…)
这个allocate_stream是Vendor实现的，对于Qualcomm的Camera，位于/path/to/aosp/hardware/qcom/camera/QCamera/HAL2/wrapper/QualcommCamera.cpp
android::allocate_stream(…)
QCameraHardwareInterface::allocate_stream(…)
QCameraStream::createInstance(…)
QCameraStream_preview::createInstance(uint32_t CameraHandle,
uint32_t ChannelId,
uint32_t Width,
uint32_t Height,
int requestedFormat,
mm_camera_vtbl_t *mm_ops,
camera_mode_t mode)
{
QCameraStream* pme = new QCameraStream_preview(CameraHandle,
ChannelId,
Width,
Height,
requestedFormat,
mm_ops,
mode);
return pme;
}
尽管这改进目前还是还是算比较“新”的一个东西，但这是趋势，所以了解下也无妨！
P.S. 4.3 在今天都发布了，我还在看4.2的东西，吼吼
网络上很多讲解媒体文件格式(比如MP4档案)的文章，有不少是作者懂了相关的东西，然后再自己总结出来的，优点就是他们抓住重点，写出注意的地方，缺点就是有时候你会觉得很突然，突然就出来一个新的概念，定义，因为他们默认你应该熟悉相应的规范。我在这里从基本点入手，写点关于媒体文件格式的东西。
首先你应该有一份ISO/IEC 14496-12(Part 12: ISO base media file format)，这个可以从http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html免费下载到。虽然是英文的，但是一些基本的东西还是不难。
当然你应该知道一些基本的概念，比如container file, codec, track等等。比如通常看到的一个MP4文件它就是一个container file，它里面装了来自codec编码后的数据，可能包括一个或者多个track(通常有一个视频track，一个音频track，有的可能有多个音频track，比如中文/英文/广东话/西班牙语，还有可能有字幕track)。
首先需要阅读最基本的几个章节就可以
3 Terms, definitions, and abbreviated terms
4 Object-structured File Organization
5 Design Considerations
即一些基本术语，定义，比如Box, Chunk, Container Box, Hint Box, Sample等等
其中文件组织结构当中规定了文件是有一系列的Box组成，在文件当中除了Box没有其他东西，所有数据都是包含在Box当中，以及为什么要采取这样面相对象的设计。
通常Box由header和包含在其中的data组成，header中包含了这个Box的size和type信息，当然size是包含了这个Box当中所有数据在内，也包含这个header。
如果Box的type不能被识别，那么这个Box就该被忽略。
aligned(8) class Box(unsigned int(32) boxtype,
optional unsigned int(8)[16] extended_type) {
unsigned int(32) size;
unsigned int(32) type = boxtype;
if (size == 1) { // 如果size为1就用largesize来表示这个Box的大小
unsigned int(64) largesize;
} else if (size == 0) {
// box extends to end of file
}
if (boxtype == 'uuid') { // 这里表示自定义的类型
unsigned int(8)[16] usertype = extended_type;
}
}
这里都是按照8位对齐，所以最小单元是8位，即一个字节，本文章中所指的单元均是这里表述的意思。
所以你就可以很容易的知道，属性size和type分别各自占用4个单元。
另外很多Box都包含version和flag这两个字段，所以又抽象出来一个FullBox
aligned(8) class FullBox(unsigned int(32) boxtype, unsigned int(8) v, bit(24) f)
extends Box(boxtype) {
unsigned int(8) version = v; // 1个单元
bit(24) flags = f; // 3个单元
}
如果Box的version不能被识别，那么这个Box就该被忽略。
所有符合这规范的文件都应当包含一个File Type Box，并且这个Box应当放置在文件尽可能早的位置，那么这是个什么样的东西？
aligned(8) class FileTypeBox
extends Box('ftyp') {
unsigned int(32) major_brand;
unsigned int(32) minor_version;
unsigned int(32) compatible_brands[]; // to end of the box // 这个是不定长的，可以是1个单元，也可以是2，什么时候结束靠前面的size决定
}
那我们还可以知道，一个典型的符合规范的文件包括moov和mdat这两个Boxes，那这又是什么？
这个时候就要继续往下看
6 ISO Base Media File organization
了，Table 1 — Box types, structure, and cross-reference这张图写出了基本上所有的Boxes，可以简单的理解就是moov(Movie Box)是包含描述信息的Box，mdat(Media Data Box)是包含真实多媒体数据的Box，而且这些Boxes都是并列或者嵌套的关系，It’s tree-like。到这里最重要的东西已经都接触了，然后你还要知道这些并列或者嵌套的Boxes要怎么摆放，也就是它的顺序，特别是moov(Movie Box)当中的子Boxes的顺序，另外因为有Streaming Support/Realtime downloading and playing的需求，moov放在mdat的前面，这样只要下载完moov数据就可以解析出来这段视频有多长，什么语言，等等信息，然后就开始播放，后面mdat数据慢慢来下载，因为mdat通常比moov大的多；因为录影功能，我们不知道什么时候录影结束，而且moov相对较小别且大小可能受到mdat大小的影响，所以我们也会把moov放到mdat后面(录影过程当中先把moov保存在内存当中，等mdat写完了再追加到尾部去)。
这是一些基本的理论，当然还有很多东西没有在这里指出，你需要阅读规范来了解，关于具体Box相关的，建议你用到这个Box的时候就去读，因为实在是太多的Boxes了。
现在我们通过一个实例来看下
guohai@KNIGHT:~$ AtomicParsley /path/to/VID_FILE.mp4 -T
Atom ftyp @ 0 of size: 24, ends @ 24
Atom moov @ 24 of size: 53884, ends @ 53908
Atom mvhd @ 32 of size: 108, ends @ 140
Atom trak @ 140 of size: 45719, ends @ 45859
Atom tkhd @ 148 of size: 92, ends @ 240
Atom mdia @ 240 of size: 45619, ends @ 45859
Atom mdhd @ 248 of size: 32, ends @ 280
Atom hdlr @ 280 of size: 44, ends @ 324
Atom minf @ 324 of size: 45535, ends @ 45859
Atom vmhd @ 332 of size: 20, ends @ 352
Atom dinf @ 352 of size: 36, ends @ 388
Atom dref @ 360 of size: 28, ends @ 388
Atom stbl @ 388 of size: 45471, ends @ 45859
Atom stsd @ 396 of size: 151, ends @ 547
Atom avc1 @ 412 of size: 135, ends @ 547
Atom avcC @ 498 of size: 33, ends @ 531
Atom pasp @ 531 of size: 16, ends @ 547 ~
Atom stts @ 547 of size: 21024, ends @ 21571
Atom stss @ 21571 of size: 232, ends @ 21803
Atom stsz @ 21803 of size: 10536, ends @ 32339
Atom stsc @ 32339 of size: 7072, ends @ 39411
Atom stco @ 39411 of size: 6448, ends @ 45859
Atom trak @ 45859 of size: 8049, ends @ 53908
Atom tkhd @ 45867 of size: 92, ends @ 45959
Atom mdia @ 45959 of size: 7949, ends @ 53908
Atom mdhd @ 45967 of size: 32, ends @ 45999
Atom hdlr @ 45999 of size: 44, ends @ 46043
Atom minf @ 46043 of size: 7865, ends @ 53908
Atom smhd @ 46051 of size: 16, ends @ 46067
Atom dinf @ 46067 of size: 36, ends @ 46103
Atom dref @ 46075 of size: 28, ends @ 46103
Atom stbl @ 46103 of size: 7805, ends @ 53908
Atom stsd @ 46111 of size: 69, ends @ 46180
Atom samr @ 46127 of size: 53, ends @ 46180
Atom damr @ 46163 of size: 17, ends @ 46180
Atom stts @ 46180 of size: 32, ends @ 46212
Atom stsz @ 46212 of size: 20, ends @ 46232
Atom stsc @ 46232 of size: 52, ends @ 46284
Atom stco @ 46284 of size: 7624, ends @ 53908
Atom free @ 53908 of size: 351116, ends @ 405024
Atom mdat @ 405024 of size: 39866486, ends @ 40271510
看到这些不要慌张，这其实就是一个MP4文件当中所有的Boxes的树形展示，我们是通过AtomicParsley把它显示出来，看看最外面有ftyp，moov，free和mdat这四个Boxes，Box在有的地方又被称为Atom，是一个意思。如果你不明白free是什么Box，那么你就需要去读下规范了。
具体看下第一行
Atom ftyp @ 0 of size: 24, ends @ 24
表示ftyp从第0个单元开始，大小是24，结束在24单元之前。
其它行以类似的方法可以读懂。
或者你可以去下载一个叫做Mp4Info的图形化工具来查看这棵由Boxes组成的树，不过这个软件只能在Windows下使用，而且有时候会crash，有些Box的Hex数据也显示不全，不过基本上使用还是可以的，查看数据可以用专门的16进制编辑器。我这里用的是一个叫做Bless的工具。
还有很多工具，可以到网上去搜索。
现在以实际例子来分析下。
我们知道第一个Box是ftyp，从0开始，大小是24。那么它的数据就是下面(用Bless打开MP4档案看到的)。
00 00 00 18 66 74 79 70 69 73 6F 6D 00 00 00 00 69 73 6F 6D 33 67 70 34
最开始4个单元是size，其值为00 00 00 18，换成decimal为24
接下来4个单元是type，其值为66 74 79 70，对应’f', ‘t’, ‘y’, ‘p’这4个字符的ASCII编码
major_brand从第9单元开始，其值为69 73 6F 6D，对应’i', ‘s’, ‘o’, ‘m’
minor_version从第13单元开始，其值为00 00 00 00，也就是0
compatible_brands从第17单元开始，包含了2组4个单元的数据，也就是8个单元，之前我们说过了，它是不定长的，大小受整个Box的大小控制，其值为69 73 6F 6D 33 67 70 34，对应’i', ‘s’, ‘o’, ‘m’, ’3′, ‘g’, ‘p’, ’4′
这就是ftyp这个Box的所有信息。
接下来我们再看下mvhd，它是moov的header，长度共计108个单元(version为0，其它版本请查看规范)，它的数据如下：
00 00 00 6C 6D 76 68 64 00 00 00 00 50 4C 2C 57 50 4C 2C 57 00 00 03 E8 00 03 7B 90 00 01 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03
规范中MovieHeaderBox的定义如下：
aligned(8) class MovieHeaderBox extends FullBox('mvhd', version, 0) {
if (version == 1) {
unsigned int(64) creation_time;
unsigned int(64) modification_time;
unsigned int(32) timescale;
unsigned int(64) duration;
} else { // version==0
unsigned int(32) creation_time;
unsigned int(32) modification_time;
unsigned int(32) timescale;
unsigned int(32) duration;
}
template int(32) rate = 0x00010000; // typically 1.0
template int(16) volume = 0x0100; // typically, full volume
const bit(16) reserved = 0;
const unsigned int(32)[2] reserved = 0;
template int(32)[9] matrix =
{ 0x00010000,0,0,0,0x00010000,0,0,0,0x40000000 }; // Unity matrix
bit(32)[6] pre_defined = 0;
unsigned int(32) next_track_ID;
}
最开始4个单元是size，其值为00 00 00 6C，换成decimal为108(6 * 16 + 12)
接下来4个单元是type，其值为6D 76 68 64，对应’m', ‘v’, ‘h’, ‘d’这4个字符的ASCII编码
接下来就是version, flag, creation_time, modification_time等等
这里有点需要注意的是，规范上说对于version为0和为1有些属性的长度是不一样的，我们这里可以来看一下。
比如这里version是9单元，它只占用一个字节，其值为00，也就是0，那么后面creation_time，modification_time，time scale和duration均每个属性占用4个单元。
creation_time从13单元开始，其值为50 4C 2C 57，这是一个timestamp，转化成string类型表示为”2012-09-09 13:42:47″
modification_time从17单元开始，其值为50 4C 2C 57，如上，所以这个档案是在2012-09-09 13:42:47创建过之后也没有修改过。
time scale从21单元开始，其值为00 00 03 E8，换算成decimal为1000
duration从25单元开始，其值为00 03 7B 90，换算成decimal为228240
该duration是所有track当中最长的那个，在规范当中有写，另外这里的duration是通过time scale表示的。
换算成我们通常的时间坐标系就是228240 / 1000 = 228.24，约为228秒，即228240毫秒。
媒体档案格式基本就是按照此类方法来分析的，比如这段视频有几个track，以及是分别是什么track，就只需要去解析它的moov/trak的Box。
遇到不是很明白的概念的时候尽可能的先到规范当中找找看有没有定义，一般解释的都还是比较清楚。比如下面这个最基本的。
[mvhd]
timescale is an integer that specifies the time-scale for the entire presentation; this is the number of time units that pass in one second. For example, a time coordinate system that measures time in sixtieths of a second has a time scale of 60.
[mvhd]
duration is an integer that declares length of the presentation (in the indicated timescale). This property is derived from the presentation’s tracks: the value of this field corresponds to the duration of the longest track in the presentation. If the duration cannot be determined then duration is set to all 1s.
[mvhd]
rate is a fixed point 16.16 number that indicates the preferred rate to play the presentation; 1.0 (0×00010000) is normal forward playback
[trak]
duration is an integer that indicates the duration of this track (in the timescale indicated in the Movie Header Box). The value of this field is equal to the sum of the durations of all of the track’s edits. If there is no edit list, then the duration is the sum of the sample durations, converted into the timescale in the Movie Header Box. If the duration of this track cannot be determined then duration is set to all 1s.
[trak]
layer specifies the front-to-back ordering of video tracks; tracks with lower numbers are closer to the viewer. 0 is the normal value, and -1 would be in front of track 0, and so on.
推荐参考的资料：
MP4文件格式解析系列(http://blog.sina.com.cn/s/blog_48f93b530100jz4b.html)
QuickTime container(http://wiki.multimedia.cx/index.php?title=QuickTime_container)
http://mp4ra.org/atoms.html
昨天晚上发生了一件悲剧的事情，写了一下午的代码在mv过程当中被删除了。事情经过是这样子的，创建好Git仓库之后，准备把代码文件mv进对应的文件夹，原本目录的样子
$ tree . |-- a |-- a.c
就这两个简单的文件，所以我直接用
$ mv a*
大家可以看到这个命令没有敲完，因为后面还缺destination directory，然后我就切换到那个目录看了一眼路径，回来的时候鬼使神差的没有把路径敲上去就让这个命令执行了，然后惨剧就发生了，这个命令没有出错，我有种预感出事情了，但是这个时候心理还是期望a.c把a给覆盖了，但是ls看了下文件大小就觉得完了，源文件被覆盖了，因为mv没有撤回的命令，所以知道麻烦来了，网络上有人说这个时候可以umount被删文件所在的分区，然后用root用户对该分区进行grep，当然关键字就是被删文件当中有的关键字，越特殊越好，我有试过这方法，但是运气没那么好，只搜出来那个文件的名字，没有内容。想想算了，自己再重写一遍吧，源文件生成的可执行程序还在，里面还有些打印的文本信息，根据这些文本信息来重新写也还行，然后花了2个多小时，到凌晨2点，终于算写完了，功能也和侥幸留下的那个可执行文件一样。于是就睡觉了。
但是我在想mv是如何解析”a*”这个字串，然后把文件覆盖掉的呢？
早上起来看了下mv程序的代码
n_files = argc - optind; // optind是排除前面program name和options之后的参数索引， // 也就是被操作的文件或者文件夹 file = argv + optind;
我们都知道argv[0]是program name，所以”a*”这种情况optind就为1，于是
$ mv a*
就被解释成了
$ mv a a.c
所以当然a.c就被a给覆盖了
那为什么mv在覆盖的时候没有给予提示，因为在Linux/Unix的世界里面，它默认你应该知道你在执行什么操作。比如这个命令，移动A文件到B文件，这是再正常正确不过的一种操作，它就是认为你是要用A把B给覆盖的。
当然”a*”变成a和a.c这纯粹是一种巧合，因为一旦你要操作的文件(夹)数目大于2，它就会去检测最后一个是否是文件夹，如果是的，就把前面的文件(夹)移动到最后一个文件夹里面，如果不是，就告诉你输入错了。
if (target_directory_operand (file[n_files - 1]))
target_directory = file[--n_files];
else if (2 < n_files)
error (EXIT_FAILURE, 0, _("target %s is not a directory"),
quote (file[n_files - 1]));
if (target_directory)
{
int i;
ok = true;
for (i = 0; i < n_files; ++i)
ok &= movefile (file[i], target_directory, true, &x);
}
else
ok = movefile (file[0], file[1], false, &x);
所以如果我当前目录有2个以上的a打头的文件，那么我执行”mv a*”，它就会给出提示，或者只是说把文件挪动了一个位置，并不会覆盖。
好了，如果你担心以后在执行cp和mv的时候文件会给覆盖的话，你可以加”-i”参数，这样碰到覆盖的情况它就会征求你的意见，当然我们这么做只是为了避免像如上这个例子的巧合和意外。你始终还是要知道你在执行什么操作。
方便起见，你可以在.bashrc当中添加如下配置就可以。
# some interactive reminders alias mv='mv -i' alias cp='cp -i'
为什么rm不加，因为你要知道rm就是要把你的资料弄没的，所以要万分小心！
当然如果你知道你要用VCS来管理你的东西，那么一开始就用吧！
我们知道在HAL的Vendor实现当中会动态去load一个名字为camera.$platform$.so的档案，然后去加载Android HAL当中定义的方法，这里以Camera HAL 2.0并且Qualcomm msm8960为例子看下，结合之前的一篇文章(http://guoh.org/lifelog/2013/07/glance-at-camera-hal-2-0/)。
(注：这篇文章已经草稿比较久了，但是一直没有发出来，因为手里的这版代码没有设备可以跑，另外也无法确定代码是否完全正确，至少发现了一些地方都是stub实现，文中可能存在一些错误，如发现不正确的地方欢迎指出，我也会尽量发现错误并修正！)
我们知道在camera2.h当中定义了很多方法，那么在msm8960 HAL就是在如下地方
/path/to/qcam-hal/QCamera/HAL2
这编译出来就是一个camera.$platform$.so，请看它的实现
首先是HAL2/wrapper/QualcommCamera.h|cpp
/**
* The functions need to be provided by the camera HAL.
*
* If getNumberOfCameras() returns N, the valid cameraId for getCameraInfo()
* and openCameraHardware() is 0 to N-1.
*/
static hw_module_methods_t camera_module_methods = {
open: camera_device_open,
};
static hw_module_t camera_common = {
tag: HARDWARE_MODULE_TAG,
module_api_version: CAMERA_MODULE_API_VERSION_2_0, // 这样Camera Service才会去初始化Camera2Client一系列
hal_api_version: HARDWARE_HAL_API_VERSION,
id: CAMERA_HARDWARE_MODULE_ID,
name: "Qcamera",
author:"Qcom",
methods: &camera_module_methods,
dso: NULL,
reserved: {0},
};
camera_module_t HAL_MODULE_INFO_SYM = { // 这个HMI，每个HAL模块都必须有的
common: camera_common,
get_number_of_cameras: get_number_of_cameras,
get_camera_info: get_camera_info,
};
camera2_device_ops_t camera_ops = { // 注意这些绑定的函数
set_request_queue_src_ops: android::set_request_queue_src_ops,
notify_request_queue_not_empty: android::notify_request_queue_not_empty,
set_frame_queue_dst_ops: android::set_frame_queue_dst_ops,
get_in_progress_count: android::get_in_progress_count,
flush_captures_in_progress: android::flush_captures_in_progress,
construct_default_request: android::construct_default_request,
allocate_stream: android::allocate_stream,
register_stream_buffers: android::register_stream_buffers,
release_stream: android::release_stream,
allocate_reprocess_stream: android::allocate_reprocess_stream,
allocate_reprocess_stream_from_stream: android::allocate_reprocess_stream_from_stream,
release_reprocess_stream: android::release_reprocess_stream,
trigger_action: android::trigger_action,
set_notify_callback: android::set_notify_callback,
get_metadata_vendor_tag_ops: android::get_metadata_vendor_tag_ops,
dump: android::dump,
};
typedef struct { // 注意这个是Qualcomm自己定义的一个wrap结构
camera2_device_t hw_dev; // 这里是标准的
QCameraHardwareInterface *hardware;
int camera_released;
int cameraId;
} camera_hardware_t;
/* HAL should return NULL if it fails to open camera hardware. */
extern "C" int camera_device_open(
const struct hw_module_t* module, const char* id,
struct hw_device_t** hw_device)
{
int rc = -1;
int mode = 0;
camera2_device_t *device = NULL;
if (module && id && hw_device) {
int cameraId = atoi(id);
if (!strcmp(module->name, camera_common.name)) {
camera_hardware_t *camHal =
(camera_hardware_t *) malloc(sizeof (camera_hardware_t));
if (!camHal) {
*hw_device = NULL;
ALOGE("%s: end in no mem", __func__);
return rc;
}
/* we have the camera_hardware obj malloced */
memset(camHal, 0, sizeof (camera_hardware_t));
camHal->hardware = new QCameraHardwareInterface(cameraId, mode);
if (camHal->hardware && camHal->hardware->isCameraReady()) {
camHal->cameraId = cameraId;
device = &camHal->hw_dev; // 这里camera2_device_t
device->common.close = close_camera_device; // 初始化camera2_device_t
device->common.version = CAMERA_DEVICE_API_VERSION_2_0;
device->ops = &camera_ops;
device->priv = (void *)camHal;
rc = 0;
} else {
if (camHal->hardware) {
delete camHal->hardware;
camHal->hardware = NULL;
}
free(camHal);
device = NULL;
}
}
}
/* pass actual hw_device ptr to framework. This amkes that we actally be use memberof() macro */
*hw_device = (hw_device_t*)&device->common; // 这就是kernel或者Android native framework常用的一招
return rc;
}
看看allocate stream
int allocate_stream(const struct camera2_device *device,
uint32_t width,
uint32_t height,
int format,
const camera2_stream_ops_t *stream_ops,
uint32_t *stream_id,
uint32_t *format_actual,
uint32_t *usage,
uint32_t *max_buffers)
{
QCameraHardwareInterface *hardware = util_get_Hal_obj(device);
hardware->allocate_stream(width, height, format, stream_ops,
stream_id, format_actual, usage, max_buffers);
return rc;
}
这里注意QCameraHardwareInterface在QCameraHWI.h|cpp当中
int QCameraHardwareInterface::allocate_stream(
uint32_t width,
uint32_t height, int format,
const camera2_stream_ops_t *stream_ops,
uint32_t *stream_id,
uint32_t *format_actual,
uint32_t *usage,
uint32_t *max_buffers)
{
int ret = OK;
QCameraStream *stream = NULL;
camera_mode_t myMode = (camera_mode_t)(CAMERA_MODE_2D|CAMERA_NONZSL_MODE);
stream = QCameraStream_preview::createInstance(
mCameraHandle->camera_handle,
mChannelId,
width,
height,
format,
mCameraHandle,
myMode);
stream->setPreviewWindow(stream_ops); // 这里，也就是只要通过该方法创建的stream，都会有对应的ANativeWindow进来
*stream_id = stream->getStreamId();
*max_buffers= stream->getMaxBuffers(); // 从HAL得到的
*usage = GRALLOC_USAGE_HW_CAMERA_WRITE | CAMERA_GRALLOC_HEAP_ID
| CAMERA_GRALLOC_FALLBACK_HEAP_ID;
/* Set to an arbitrary format SUPPORTED by gralloc */
*format_actual = HAL_PIXEL_FORMAT_YCrCb_420_SP;
return ret;
}
QCameraStream_preview::createInstance直接调用自己的构造方法，也就是下面
(相关class在QCameraStream.h|cpp和QCameraStream_Preview.cpp)
QCameraStream_preview::QCameraStream_preview(uint32_t CameraHandle,
uint32_t ChannelId,
uint32_t Width,
uint32_t Height,
int requestedFormat,
mm_camera_vtbl_t *mm_ops,
camera_mode_t mode) :
QCameraStream(CameraHandle,
ChannelId,
Width,
Height,
mm_ops,
mode),
mLastQueuedFrame(NULL),
mDisplayBuf(NULL),
mNumFDRcvd(0)
{
mStreamId = allocateStreamId(); // 分配stream id(根据mStreamTable)
switch (requestedFormat) { // max buffer number
case CAMERA2_HAL_PIXEL_FORMAT_OPAQUE:
mMaxBuffers = 5;
break;
case HAL_PIXEL_FORMAT_BLOB:
mMaxBuffers = 1;
break;
default:
ALOGE("Unsupported requested format %d", requestedFormat);
mMaxBuffers = 1;
break;
}
/*TODO: There has to be a better way to do this*/
}
再看看
/path/to/qcam-hal/QCamera/stack/mm-camera-interface/
mm_camera_interface.h
当中
typedef struct {
uint32_t camera_handle; /* camera object handle */
mm_camera_info_t *camera_info; /* reference pointer of camear info */
mm_camera_ops_t *ops; /* API call table */
} mm_camera_vtbl_t;
mm_camera_interface.c
当中
/* camera ops v-table */
static mm_camera_ops_t mm_camera_ops = {
.sync = mm_camera_intf_sync,
.is_event_supported = mm_camera_intf_is_event_supported,
.register_event_notify = mm_camera_intf_register_event_notify,
.qbuf = mm_camera_intf_qbuf,
.camera_close = mm_camera_intf_close,
.query_2nd_sensor_info = mm_camera_intf_query_2nd_sensor_info,
.is_parm_supported = mm_camera_intf_is_parm_supported,
.set_parm = mm_camera_intf_set_parm,
.get_parm = mm_camera_intf_get_parm,
.ch_acquire = mm_camera_intf_add_channel,
.ch_release = mm_camera_intf_del_channel,
.add_stream = mm_camera_intf_add_stream,
.del_stream = mm_camera_intf_del_stream,
.config_stream = mm_camera_intf_config_stream,
.init_stream_bundle = mm_camera_intf_bundle_streams,
.destroy_stream_bundle = mm_camera_intf_destroy_bundle,
.start_streams = mm_camera_intf_start_streams,
.stop_streams = mm_camera_intf_stop_streams,
.async_teardown_streams = mm_camera_intf_async_teardown_streams,
.request_super_buf = mm_camera_intf_request_super_buf,
.cancel_super_buf_request = mm_camera_intf_cancel_super_buf_request,
.start_focus = mm_camera_intf_start_focus,
.abort_focus = mm_camera_intf_abort_focus,
.prepare_snapshot = mm_camera_intf_prepare_snapshot,
.set_stream_parm = mm_camera_intf_set_stream_parm,
.get_stream_parm = mm_camera_intf_get_stream_parm
};
以start stream为例子
mm_camera_intf_start_streams(mm_camera_interface
mm_camera_start_streams(mm_camera
mm_channel_fsm_fn(mm_camera_channel
mm_channel_fsm_fn_active(mm_camera_channel
mm_channel_start_streams(mm_camera_channel
mm_stream_fsm_fn(mm_camera_stream
mm_stream_fsm_reg(mm_camera_stream
mm_camera_cmd_thread_launch(mm_camera_data
mm_stream_streamon(mm_camera_stream
注意：本文当中，如上这种梯度摆放，表示是调用关系，如果梯度是一样的，就表示这些方法是在上层同一个方法里面被调用的
int32_t mm_stream_streamon(mm_stream_t *my_obj)
{
int32_t rc;
enum v4l2_buf_type buf_type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
/* Add fd to data poll thread */
rc = mm_camera_poll_thread_add_poll_fd(&my_obj->ch_obj->poll_thread[0],
my_obj->my_hdl,
my_obj->fd,
mm_stream_data_notify,
(void*)my_obj);
if (rc < 0) {
return rc;
}
rc = ioctl(my_obj->fd, VIDIOC_STREAMON, &buf_type);
if (rc < 0) {
CDBG_ERROR("%s: ioctl VIDIOC_STREAMON failed: rc=%d\n",
__func__, rc);
/* remove fd from data poll thread in case of failure */
mm_camera_poll_thread_del_poll_fd(&my_obj->ch_obj->poll_thread[0], my_obj->my_hdl);
}
return rc;
}
看到ioctl，VIDIOC_STREAMON，可以高兴一下了，这就是V4L2规范当中用户空间和内核空间通信的方法，V4L2(Video for Linux Two)是一种经典而且成熟的视频通信协议，之前是V4L，不清楚的可以去下载它的规范，另外The Video4Linux2(http://lwn.net/Articles/203924/)也是很好的资料。
这里简单介绍下：
open(VIDEO_DEVICE_NAME, …) // 开启视频设备，一般在程序初始化的时候调用
ioctl(…) // 主要是一些需要传输数据量很小的控制操作
这里可以用的参数很多，并且通常来说我们会按照以下方式来使用，比如
VIDIOC_QUERYCAP // 查询设备能干什么
VIDIOC_CROPCAP // 查询设备crop能力
VIDIOC_S_* // set/get方法，设置/获取参数
VIDIOC_G_*
VIDIOC_REQBUFS // 分配buffer，可以有多种方式
VIDIOC_QUERYBUF // 查询分配的buffer的信息
VIDIOC_QBUF // QUEUE BUFFER 把buffer压入DRV缓存队列(这时候buffer是空的)
VIDIOC_STREAMON // 开始视频数据传输
VIDIOC_DQBUF // DEQUEUE BUFFER 把buffer从DRV缓存队列中取出(这时候buffer是有数据的)[0...n]
QBUF -> DQBUF // 可以一直重复这个动作VIDIOC_STREAMOFF // 停止视频数据传输
close(VIDEO_DEVICE_FD) // 关闭设备
上面就是主要的函数和简单的调用顺序，另外还有几个函数select() // 等待事件发生，主要用在我们把存frame的buffer推给DRV以后，等待它的反应
mmap/munmap // 主要处理我们request的buffer的，buffer分配在设备的内存空间的时候需要
并且看看mm_camera_stream这个文件里面也都是这么实现的。
看完这里，我们回过头来继续看QCam HAL，当然它实现的细节也不是我上面start stream所列的那么简单，但是其实也不算复杂，觉得重要的就是状态和用到的结构。
首先是channel状态，目前只支持1个channel，但是可以有多个streams(后面会介绍，而且目前最多支持8个streams)
/* mm_channel */
typedef enum {
MM_CHANNEL_STATE_NOTUSED = 0, /* not used */
MM_CHANNEL_STATE_STOPPED, /* stopped */
MM_CHANNEL_STATE_ACTIVE, /* active, at least one stream active */
MM_CHANNEL_STATE_PAUSED, /* paused */
MM_CHANNEL_STATE_MAX
} mm_channel_state_type_t;
它可以执行的事件
typedef enum {
MM_CHANNEL_EVT_ADD_STREAM,
MM_CHANNEL_EVT_DEL_STREAM,
MM_CHANNEL_EVT_START_STREAM,
MM_CHANNEL_EVT_STOP_STREAM,
MM_CHANNEL_EVT_TEARDOWN_STREAM,
MM_CHANNEL_EVT_CONFIG_STREAM,
MM_CHANNEL_EVT_PAUSE,
MM_CHANNEL_EVT_RESUME,
MM_CHANNEL_EVT_INIT_BUNDLE,
MM_CHANNEL_EVT_DESTROY_BUNDLE,
MM_CHANNEL_EVT_REQUEST_SUPER_BUF,
MM_CHANNEL_EVT_CANCEL_REQUEST_SUPER_BUF,
MM_CHANNEL_EVT_START_FOCUS,
MM_CHANNEL_EVT_ABORT_FOCUS,
MM_CHANNEL_EVT_PREPARE_SNAPSHOT,
MM_CHANNEL_EVT_SET_STREAM_PARM,
MM_CHANNEL_EVT_GET_STREAM_PARM,
MM_CHANNEL_EVT_DELETE,
MM_CHANNEL_EVT_MAX
} mm_channel_evt_type_t;
/* mm_stream */
typedef enum { // 这里的状态要仔细，每执行一次方法，状态就需要变化
MM_STREAM_STATE_NOTUSED = 0, /* not used */
MM_STREAM_STATE_INITED, /* inited */
MM_STREAM_STATE_ACQUIRED, /* acquired, fd opened */
MM_STREAM_STATE_CFG, /* fmt & dim configured */
MM_STREAM_STATE_BUFFED, /* buf allocated */
MM_STREAM_STATE_REG, /* buf regged, stream off */
MM_STREAM_STATE_ACTIVE_STREAM_ON, /* active with stream on */
MM_STREAM_STATE_ACTIVE_STREAM_OFF, /* active with stream off */
MM_STREAM_STATE_MAX
} mm_stream_state_type_t;
同样，stream可以执行的事件
typedef enum {
MM_STREAM_EVT_ACQUIRE,
MM_STREAM_EVT_RELEASE,
MM_STREAM_EVT_SET_FMT,
MM_STREAM_EVT_GET_BUF,
MM_STREAM_EVT_PUT_BUF,
MM_STREAM_EVT_REG_BUF,
MM_STREAM_EVT_UNREG_BUF,
MM_STREAM_EVT_START,
MM_STREAM_EVT_STOP,
MM_STREAM_EVT_QBUF,
MM_STREAM_EVT_SET_PARM,
MM_STREAM_EVT_GET_PARM,
MM_STREAM_EVT_MAX
} mm_stream_evt_type_t;
这里每次执行函数的时候都需要检查channel/stream的状态，只有状态正确的时候才会去执行
比如你可以观察到
mm_channel的mm_channel_state_type_t state;
mm_stream的mm_stream_state_type_t state;
均表示这个结构当前的状态
另外
struct mm_camera_obj
struct mm_channel
struct mm_stream
这三个也是自上而下包含的，并且stream和channel还会持有父结构(暂且这么称呼，实际为container关系)的引用。
实际上Vendor的HAL每个都有自己实现的方法，也可能包含很多特有的东西，比如这里它会喂给ioctl一些特有的命令或者数据结构，这些我们就只有在做特定平台的时候去考虑了。这些都可能千变万化，比如OMAP4它同DRV沟通是透过rpmsg，并用OpenMAX的一套规范来实现的。
理论就这么多，接着看一个实例，比如我们在Camera Service要去start preview：
Camera2Client::startPreviewL StreamingProcessor->updatePreviewStream Camera2Device->createStream StreamAdapter->connectToDevice camera2_device_t->ops->allocate_stream // 上面有分析 native_window_api_*或者native_window_* StreamingProcessor->startStream Camera2Device->setStreamingRequest Camera2Device::RequestQueue->setStreamSlot // 创建一个stream slot Camera2Device::RequestQueue->signalConsumerLocked
status_t Camera2Device::MetadataQueue::signalConsumerLocked() {
status_t res = OK;
notEmpty.signal();
if (mSignalConsumer && mDevice != NULL) {
mSignalConsumer = false;
mMutex.unlock();
res = mDevice->ops->notify_request_queue_not_empty(mDevice); // 通知Vendor HAL的run command thread去运行，
// notify_request_queue_not_empty这个事件不是每次都会触发的，只有初始化时候
// 或者run command thread在dequeue的时候发现数据为NULL，
// 而Camera Service之变又有新的request进来的时候才会去触发
// 可以说是减轻负担吧，不用没有请求的时候，thread也一直在那里
// 不过通常碰到这样的情况都是利用锁让thread停在那里
mMutex.lock();
}
return res;
}
然而在Qualcomm HAL当中
int notify_request_queue_not_empty(const struct camera2_device *device) // 这个方法注册到camera2_device_ops_t当中 QCameraHardwareInterface->notify_request_queue_not_empty() pthread_create(&mCommandThread, &attr, command_thread, (void *)this) != 0)
void *command_thread(void *obj)
{
...
pme->runCommandThread(obj);
}
void QCameraHardwareInterface::runCommandThread(void *data)
{
/**
* This function implements the main service routine for the incoming
* frame requests, this thread routine is started everytime we get a
* notify_request_queue_not_empty trigger, this thread makes the
* assumption that once it receives a NULL on a dequest_request call
* there will be a fresh notify_request_queue_not_empty call that is
* invoked thereby launching a new instance of this thread. Therefore,
* once we get a NULL on a dequeue request we simply let this thread die
*/
int res;
camera_metadata_t *request=NULL;
mPendingRequests=0;
while (mRequestQueueSrc) { // mRequestQueueSrc是通过set_request_queue_src_ops设置进来的
// 参见Camera2Device::MetadataQueue::setConsumerDevice
// 在Camera2Device::initialize当中被调用
ALOGV("%s:Dequeue request using mRequestQueueSrc:%p",__func__,mRequestQueueSrc);
mRequestQueueSrc->dequeue_request(mRequestQueueSrc, &request); // 取framework request
if (request==NULL) {
ALOGE("%s:No more requests available from src command \
thread dying",__func__);
return;
}
mPendingRequests++;
/* Set the metadata values */
/* Wait for the SOF for the new metadata values to be applied */
/* Check the streams that need to be active in the stream request */
sort_camera_metadata(request);
camera_metadata_entry_t streams;
res = find_camera_metadata_entry(request,
ANDROID_REQUEST_OUTPUT_STREAMS,
&streams);
if (res != NO_ERROR) {
ALOGE("%s: error reading output stream tag", __FUNCTION__);
return;
}
res = tryRestartStreams(streams); // 会去prepareStream和streamOn，后面有详细代码
if (res != NO_ERROR) {
ALOGE("error tryRestartStreams %d", res);
return;
}
/* 3rd pass: Turn on all streams requested */
for (uint32_t i = 0; i < streams.count; i++) {
int streamId = streams.data.u8[i];
QCameraStream *stream = QCameraStream::getStreamAtId(streamId);
/* Increment the frame pending count in each stream class */
/* Assuming we will have the stream obj in had at this point may be
* may be multiple objs in which case we loop through array of streams */
stream->onNewRequest();
}
ALOGV("%s:Freeing request using mRequestQueueSrc:%p",__func__,mRequestQueueSrc);
/* Free the request buffer */
mRequestQueueSrc->free_request(mRequestQueueSrc,request);
mPendingRequests--;
ALOGV("%s:Completed request",__func__);
}
QCameraStream::streamOffAll();
}
下面这个方法解释mRequestQueueSrc来自何处
// Connect to camera2 HAL as consumer (input requests/reprocessing)
status_t Camera2Device::MetadataQueue::setConsumerDevice(camera2_device_t *d) {
ATRACE_CALL();
status_t res;
res = d->ops->set_request_queue_src_ops(d,
this);
if (res != OK) return res;
mDevice = d;
return OK;
}
因为
QCameraStream_preview->prepareStream QCameraStream->initStream mm_camera_vtbl_t->ops->add_stream(... stream_cb_routine ...) // 这是用来返回数据的callback，带mm_camera_super_buf_t*和void*两参数 mm_camera_add_stream mm_channel_fsm_fn(..., MM_CHANNEL_EVT_ADD_STREAM, ..., mm_evt_paylod_add_stream_t) mm_channel_fsm_fn_stopped mm_channel_add_stream(..., mm_camera_buf_notify_t, ...) mm_stream_fsm_inited
而
在mm_channel_add_stream当中有把mm_camera_buf_notify_t包装到mm_stream_t
mm_stream_t *stream_obj = NULL; /* initialize stream object */ memset(stream_obj, 0, sizeof(mm_stream_t)); /* cd through intf always palced at idx 0 of buf_cb */ stream_obj->buf_cb[0].cb = buf_cb; // callback stream_obj->buf_cb[0].user_data = user_data; stream_obj->buf_cb[0].cb_count = -1; /* infinite by default */ // 默认无限次数
并且mm_stream_fsm_inited，传进来的event参数也是MM_STREAM_EVT_ACQUIRE
int32_t mm_stream_fsm_inited(mm_stream_t *my_obj,
mm_stream_evt_type_t evt,
void * in_val,
void * out_val)
{
int32_t rc = 0;
char dev_name[MM_CAMERA_DEV_NAME_LEN];
switch (evt) {
case MM_STREAM_EVT_ACQUIRE:
if ((NULL == my_obj->ch_obj) || (NULL == my_obj->ch_obj->cam_obj)) {
CDBG_ERROR("%s: NULL channel or camera obj\n", __func__);
rc = -1;
break;
}
snprintf(dev_name, sizeof(dev_name), "/dev/%s",
mm_camera_util_get_dev_name(my_obj->ch_obj->cam_obj->my_hdl));
my_obj->fd = open(dev_name, O_RDWR | O_NONBLOCK); // 打开视频设备
if (my_obj->fd <= 0) {
CDBG_ERROR("%s: open dev returned %d\n", __func__, my_obj->fd);
rc = -1;
break;
}
rc = mm_stream_set_ext_mode(my_obj);
if (0 == rc) {
my_obj->state = MM_STREAM_STATE_ACQUIRED; // mm_stream_state_type_t
} else {
/* failed setting ext_mode
* close fd */
if(my_obj->fd > 0) {
close(my_obj->fd);
my_obj->fd = -1;
}
break;
}
rc = get_stream_inst_handle(my_obj);
if(rc) {
if(my_obj->fd > 0) {
close(my_obj->fd);
my_obj->fd = -1;
}
}
break;
default:
CDBG_ERROR("%s: Invalid evt=%d, stream_state=%d",
__func__,evt,my_obj->state);
rc = -1;
break;
}
return rc;
}
还有
QCameraStream->streamOn mm_camera_vtbl_t->ops->start_streams mm_camera_intf_start_streams mm_camera_start_streams mm_channel_fsm_fn(..., MM_CHANNEL_EVT_START_STREAM, ...) mm_stream_fsm_fn(..., MM_STREAM_EVT_START, ...) mm_camera_cmd_thread_launch // 启动CB线程 mm_stream_streamon(mm_stream_t) mm_camera_poll_thread_add_poll_fd(..., mm_stream_data_notify , ...)
而
static void mm_stream_data_notify(void* user_data)
{
mm_stream_t *my_obj = (mm_stream_t*)user_data;
int32_t idx = -1, i, rc;
uint8_t has_cb = 0;
mm_camera_buf_info_t buf_info;
if (NULL == my_obj) {
return;
}
if (MM_STREAM_STATE_ACTIVE_STREAM_ON != my_obj->state) {
/* this Cb will only received in active_stream_on state
* if not so, return here */
CDBG_ERROR("%s: ERROR!! Wrong state (%d) to receive data notify!",
__func__, my_obj->state);
return;
}
memset(&buf_info, 0, sizeof(mm_camera_buf_info_t));
pthread_mutex_lock(&my_obj->buf_lock);
rc = mm_stream_read_msm_frame(my_obj, &buf_info); // 通过ioctl(..., VIDIOC_DQBUF, ...)读取frame数据
if (rc != 0) {
pthread_mutex_unlock(&my_obj->buf_lock);
return;
}
idx = buf_info.buf->buf_idx;
/* update buffer location */
my_obj->buf_status[idx].in_kernel = 0;
/* update buf ref count */
if (my_obj->is_bundled) {
/* need to add into super buf since bundled, add ref count */
my_obj->buf_status[idx].buf_refcnt++;
}
for (i=0; i < MM_CAMERA_STREAM_BUF_CB_MAX; i++) {
if(NULL != my_obj->buf_cb[i].cb) {
/* for every CB, add ref count */
my_obj->buf_status[idx].buf_refcnt++;
has_cb = 1;
}
}
pthread_mutex_unlock(&my_obj->buf_lock);
mm_stream_handle_rcvd_buf(my_obj, &buf_info); // mm_camera_queue_enq，往queue里面丢frame数据(
// 前提是有注册callback)，并透过sem_post通知queue
// 然后mm_camera_cmd_thread_launch启动的线程会
// 轮循读取数据，然后执行CB
}
这样就会导致在stream on的时候stream_cb_routine(实现在QCameraStream当中)就会一直执行
void stream_cb_routine(mm_camera_super_buf_t *bufs,
void *userdata)
{
QCameraStream *p_obj=(QCameraStream*) userdata;
switch (p_obj->mExtImgMode) { // 这个mode在prepareStream的时候就会确定
case MM_CAMERA_PREVIEW:
ALOGE("%s : callback for MM_CAMERA_PREVIEW", __func__);
((QCameraStream_preview *)p_obj)->dataCallback(bufs); // CAMERA_PREVIEW和CAMERA_VIDEO是一样的？
break;
case MM_CAMERA_VIDEO:
ALOGE("%s : callback for MM_CAMERA_VIDEO", __func__);
((QCameraStream_preview *)p_obj)->dataCallback(bufs);
break;
case MM_CAMERA_SNAPSHOT_MAIN:
ALOGE("%s : callback for MM_CAMERA_SNAPSHOT_MAIN", __func__);
p_obj->p_mm_ops->ops->qbuf(p_obj->mCameraHandle,
p_obj->mChannelId,
bufs->bufs[0]);
break;
case MM_CAMERA_SNAPSHOT_THUMBNAIL:
break;
default:
break;
}
}
void QCameraStream::dataCallback(mm_camera_super_buf_t *bufs)
{
if (mPendingCount != 0) { // 这个dataCallback是一直在都在回来么？
// 而且从代码来看设置下去的callback次数默认是-1，-1就表示infinite。
// 似乎只能这样才能解释，否则没人触发的话，即使mPendingCount在onNewRequest当中加1了
// 这里也感知不到
ALOGD("Got frame request");
pthread_mutex_lock(&mFrameDeliveredMutex);
mPendingCount--;
ALOGD("Completed frame request");
pthread_cond_signal(&mFrameDeliveredCond);
pthread_mutex_unlock(&mFrameDeliveredMutex);
processPreviewFrame(bufs);
} else {
p_mm_ops->ops->qbuf(mCameraHandle,
mChannelId, bufs->bufs[0]); // 如果没有需要数据的情况，直接把buffer压入DRV的队列当中，会call到V4L2的QBUF
}
}
比较好奇的是在手里这版QCam HAL的code当中camera2_frame_queue_dst_ops_t没有被用到
int QCameraHardwareInterface::set_frame_queue_dst_ops(
const camera2_frame_queue_dst_ops_t *frame_dst_ops)
{
mFrameQueueDst = frame_dst_ops; // 这个现在似乎没有用到嘛
return OK;
}
这样Camera Service的FrameProcessor的Camera2Device->getNextFrame就永远也获取不到数据，不知道是不是我手里的这版代码的问题，而且在最新的Qualcomm Camera HAL代码也不在AOSP树当中了，而是直接以proprietary形式给的so档，这只是题外话。
所以总体来看，这里可能有几个QCameraStream，每个stream负责自己的事情。
他们之间也有相互关系，比如有可能新的stream进来会导致其他已经stream-on的stream重新启动。
在Camera HAL 2.0当中我们还有个重点就是re-process stream
简单的说就是把output stream作为input stream再次添加到BufferQueue中，让其他的consumer来处理，就类似一个chain一样。
目前在ZslProcessor当中有用到。
ZslProcessor->updateStream Camera2Device->createStream Camera2Device->createReprocessStreamFromStream // release的时候是先delete re-process new ReprocessStreamAdapter ReprocessStreamAdapter->connectToDevice camera2_device_t->ops->allocate_reprocess_stream_from_stream
这里ReprocessStreamAdapter实际就是camera2_stream_in_ops_t，负责管理re-process的stream。
但是这版的代码Qualcomm也似乎没有去实现，所以暂时到此为止，如果后面找到相应的代码，再来看。
所以看完这么多不必觉得惊讶，站在Camera Service的立场，它持有两个MetadataQueue，mRequestQueue和mFrameQueue。
app请求的动作，比如set parameter/start preview/start recording会直接转化为request，放到mRequestQueue，然后去重启preview/recording stream。
比如capture也会转换为request，放到mRequestQueue。
如果有必要，会通过notify_request_queue_not_empty去通知QCam HAL有请求需要处理，然后QCam HAL会启动一个线程(QCameraHardwareInterface::runCommandThread)去做处理。直到所有request处理完毕退出线程。
在这个处理的过程当中会分别调用到每个stream的processPreviewFrame，有必要的话它每个都会调用自己后续的callback。
还有一个实现的细节就是，stream_cb_routine是从start stream就有开始注册在同一个channel上的，而stream_cb_routine间接调用QCameraStream::dataCallback(当然stream_cb_routine有去指定这个callback回来的原因是什么，就好调用对应的dataCallback)，这个callback是一直都在回来，所以每次new request让mPendingCount加1之后，dataCallback回来才会调用processPreviewFrame，否则就直接把buffer再次压回DRV队列当中。
void QCameraStream::dataCallback(mm_camera_super_buf_t *bufs)
{
if (mPendingCount != 0) { // 这个dataCallback是一直在都在回来么？
// 而且从代码来看设置下去的callback次数默认是-1，-1就表示infinite。
// 似乎只能这样才能解释，否则没人触发的话，即使mPendingCount在onNewRequest当中加1了
// 这里也感知不到
ALOGD("Got frame request");
pthread_mutex_lock(&mFrameDeliveredMutex);
mPendingCount--;
ALOGD("Completed frame request");
pthread_cond_signal(&mFrameDeliveredCond);
pthread_mutex_unlock(&mFrameDeliveredMutex);
processPreviewFrame(bufs);
} else {
p_mm_ops->ops->qbuf(mCameraHandle,
mChannelId, bufs->bufs[0]); // 如果没有需要数据的情况，直接把buffer压入DRV的队列当中，会call到V4L2的QBUF
}
}
void QCameraStream::onNewRequest()
{
ALOGI("%s:E",__func__);
pthread_mutex_lock(&mFrameDeliveredMutex);
ALOGI("Sending Frame request");
mPendingCount++;
pthread_cond_wait(&mFrameDeliveredCond, &mFrameDeliveredMutex); // 等带一个请求处理完，再做下一个请求
ALOGV("Got frame");
pthread_mutex_unlock(&mFrameDeliveredMutex);
ALOGV("%s:X",__func__);
}
processPreviewFrame会调用到创建这个stream的时候关联进来的那个BufferQueue的enqueue_buffer方法，把数据塞到BufferQueue中，然后对应的consumer就会收到了。
比如在Android Camera HAL 2.0当中目前有
camera2/BurstCapture.h
camera2/CallbackProcessor.h
camera2/JpegProcessor.h
camera2/StreamingProcessor.h
camera2/ZslProcessor.h
实现了对应的Consumer::FrameAvailableListener，但是burst-capture现在可以不考虑，因为都还只是stub实现。
ZslProcessor.h和CaptureSequencer.h都有去实现FrameProcessor::FilteredListener的onFrameAvailable(…)
但是我们之前讲过这版QCam HAL没有实现，所以FrameProcessor是无法获取到meta data的。
所以这样来看onFrameAbailable都不会得到通知。(我相信是我手里的这版代码的问题啦)
之前我们说过QCam HAL有部分东西没有实现，所以mFrameQueue就不会有数据，但是它本来应该是DRV回来的元数据会queue到这里面。
另外
CaptureSequencer.h还有去实现onCaptureAvailable，当JpegProcessor处理完了会通知它。
好奇？多个stream(s)不是同时返回的，这样如果CPU处理快慢不同就会有时间差？还有很好奇DRV是如何处理Video snapshot的，如果buffer是顺序的，就会存在Video少一个frame，如果不是顺序的，那就是DRV一次返回多个buffer？以前真没有想过这个问题@_@
针对Jelly Bean版本的代码。
SurfaceFlinger是什么，这些介绍大家可以在网络上找找看，这里就直接上代码。
首先我们得了解一种常用的编程做法，生产者/消费者模型，也许都会觉得很简单，但是这里就用到了很多这些基本概念。
BufferQueue 数据都queue到这里面，前提是它是先从BufferQueue取出一个空的数据单元，称为一个buffer，实际为GraphicBuffer类型。
ConsumerBase 它是消费者端使用的接口，它实现了BufferQueue::ConsumerListener接口，也就是BufferQueue当中有buffer被queue的时候，它能被通知到(onFrameAvailable)。同理当生产者disconnect与BufferQueue的连接或者setBufferCount被调用(该方法释放掉所有buffer，让buffer都归BufferQueue所有，如果有buffer处于DEQUEUED状态，此方法返回错误)，它也会被通知到(onBuffersReleased)。
BufferItemConsumer和CpuConsumer 它们都是ConsumerBase的子类，BufferItemConsumer一次可以acquire多个buffer，ConsumerBase一次只能一个，BufferItemConsumer是修改了BufferQueue的mMaxAcquiredBufferCount参数，ConsumerBase使用的默认值1。CpuBuffer可以把buffer锁起来供CPU使用，它也是调用GRALLOC的方法来完成这个功能的。
FramebufferSurface ConsumerBase的子类，会把收到的数据通过HWComposer往荧幕上贴。
SurfaceTexture ConsumerBase的子类，它可以把GraphicBuffer转换成texture image，然后交给OpenGL。
SurfaceTextureLayer是一个定制化的BufferQueue，NATIVE_WINDOW_API_MEDIA/NATIVE_WINDOW_API_CAMERA过来的请求会把BufferQueue设置为异步模式。
BufferQueue当中buffer的状态，这个很简单，但是也很重要。
// BufferState represents the different states in which a buffer slot
// can be.
enum BufferState {
// FREE indicates that the buffer is not currently being used and
// will not be used in the future until it gets dequeued and
// subsequently queued by the client.
// aka "owned by BufferQueue, ready to be dequeued"
FREE = 0,
// DEQUEUED indicates that the buffer has been dequeued by the
// client, but has not yet been queued or canceled. The buffer is
// considered 'owned' by the client, and the server should not use
// it for anything.
//
// Note that when in synchronous-mode (mSynchronousMode == true),
// the buffer that's currently attached to the texture may be
// dequeued by the client. That means that the current buffer can
// be in either the DEQUEUED or QUEUED state. In asynchronous mode,
// however, the current buffer is always in the QUEUED state.
// aka "owned by producer, ready to be queued"
DEQUEUED = 1,
// QUEUED indicates that the buffer has been queued by the client,
// and has not since been made available for the client to dequeue.
// Attaching the buffer to the texture does NOT transition the
// buffer away from the QUEUED state. However, in Synchronous mode
// the current buffer may be dequeued by the client under some
// circumstances. See the note about the current buffer in the
// documentation for DEQUEUED.
// aka "owned by BufferQueue, ready to be acquired"
QUEUED = 2,
// aka "owned by consumer, ready to be released"
ACQUIRED = 3
};
BufferQueue主要方法
dequeueBuffer取一个buffer(返回slot，这个bufer是从State为FREE的当中取的)给client使用，必要时候(null/height/width/format/usage任何一点不满足都会触发)它会使用GraphicBufferAlloc::createGraphicBuffer()去分配buffer
requestBuffer根据一个指定的slot获取它的buffer的地址，这个主要用在刚刚分配buffer之后(或者是意外的发现指定slot的buffer地址为空)，目前在SurfaceTextureClient(Surface)当中被使用到
queueBuffer通知BufferQueue压入了一个装满数据的buffer，QueueBufferInput是该buffer的描述数据，QueueBufferOutput是BufferQueue当前的状态(默认height/width/transformHint/slot的数量，这个slot只是当前被还回给BufferQueue)
acquireBuffer获取一个pending buffer的拥有权，这个buffer是mQueue当中，也就是状态为QUEUED的(有没有数据？)。
releaseBuffer放弃持有的指定slot的buffer
freeBuffer或者cancelBuffer都会导致这个buffer处于FREE状态
ConsumerBase的主要方法
acquireBufferLocked/releaseBufferLocked/freeBufferLocked/abandonLocked
另外这个protected的数组也很重要，子类可以直接从它里面获取buffer的信息，它实际就相当于缓存了BufferQueue的一些必要信息。
// mSlots stores the buffers that have been allocated by the BufferQueue // for each buffer slot. It is initialized to null pointers, and gets // filled in with the result of BufferQueue::acquire when the // client dequeues a buffer from a // slot that has not yet been used. The buffer allocated to a slot will also // be replaced if the requested buffer usage or geometry differs from that // of the buffer allocated to a slot. Slot mSlots[BufferQueue::NUM_BUFFER_SLOTS];
SurfaceTextureClient是一个ANativeWindow，为native_window_api_*和native_window_*方法(这些都在system/core/include/system/window.h当中)做具体实现，另外它还持有SurfaceTexture。
// Initialize the ANativeWindow function pointers. ANativeWindow::setSwapInterval = hook_setSwapInterval; ANativeWindow::dequeueBuffer = hook_dequeueBuffer; ANativeWindow::cancelBuffer = hook_cancelBuffer; ANativeWindow::queueBuffer = hook_queueBuffer; ANativeWindow::query = hook_query; ANativeWindow::perform = hook_perform; ANativeWindow::dequeueBuffer_DEPRECATED = hook_dequeueBuffer_DEPRECATED; ANativeWindow::cancelBuffer_DEPRECATED = hook_cancelBuffer_DEPRECATED; ANativeWindow::lockBuffer_DEPRECATED = hook_lockBuffer_DEPRECATED; ANativeWindow::queueBuffer_DEPRECATED = hook_queueBuffer_DEPRECATED; const_cast<int&>(ANativeWindow::minSwapInterval) = 0; const_cast<int&>(ANativeWindow::maxSwapInterval) = 1;
一些重要的命名改动
早期的Jelly Bean当中，比如(4.1/4.2) 4.3
================================================================================
SurfaceTextureClient和Surface(继承 被简化成了Surface(ANativeWindow)
自SurfaceTextureClient)实际就是一个
ANativeWindow
================================================================================
ISurfaceTexture IGraphicBufferProducer，
Binder IPC接口，用来在不同组件之间
传输数据使用(跨进程的)，BufferQueue
实现了BnGraphicBufferProducer
================================================================================
SurfaceTexture(ConsumerBase) GLConsumer(ConsumerBase)它取
BufferQueue里面的数据，然后作为一个
texture提供给OpenGL使用
上面是会用到的基本知识，下面基本才直接和SurfaceFlinger相关。
箭头的方向为继承的方向
BpSurface ---->>>> ISurface
sp<ISurfaceTexture> ISurface::getSurfaceTexture()
BSurface ---->>>> BnSurface ---->>>> ISurface
sp<ISurfaceTexture> BSurface::getSurfaceTexture()
SurfaceTexture::getBufferQueue()
Layer ---->>>> LayerBaseClient ---->>>> LayerBase
sp<ISurface> Layer::createSurface()
new BSurface
sp<ISurface> LayerBaseClient::getSurface()
sp<ISurface> LayerBaseClient::createSurface()
BpSurfaceComposerClient ---->>>> ISurfaceComposerClient
sp<ISurface> ISurfaceComposerClient::createSurface()
Client ---->>>> BnSurfaceComposerClient ---->>>> ISurfaceComposerClient
Client::createSurface()
SurfaceFlinger::createLayer()
createXXXLayer()
new LayerXXX
Layer::getSurface()
Layer::createSurface()
sp<SurfaceControl> SurfaceComposerClient::createSurface()
ISurfaceComposerClient::createSurface()
new SurfaceControl(ISurface)
// SurfaceComposerClient只是个普通的工具类，它的createSurface会去调用ISurfaceComposerClient和createSurface
现在来看一种情况，假设客户端要创建一个SurfaceView，这中间会发生什么样的事情。
当然你先得了解在Java层当中SurfaceView/SurfaceHolder/Surface这三者是什么关系。
=================================Java===================================================== new SurfaceView surface = new Surface // 这个是SurfaceView当中的Surface(这都是空的，不会在服务端真正的去创建一个Surface) newSurface = new Surface // 这个是新的Surface，当Surface改变/被创建/被销毁/需要重绘， // 都会是现在系统层准备好，然后再复制来替代我们SurfaceView当 // 中的原来的Surface(通过transferFrom完成)
真正创建Surface的方法是系统去调用的，app不会直接去调用，但是一旦被调用之后就会进入到JNI层相应方法之中，
会用到一个SurfaceSession，书面解释是表示到Surface Flinger的一次会话，因为客户端要同服务端沟通，就存在这样一个会话的概念，这个实际就是Native层SurfaceComposerClient的一个实例。
=================================JNI&Native======================================== android_view_Surface.cpp nativeCreate() android_view_SurfaceSession_getClient SurfaceComposerClient->createSurface ISurfaceComposerClient->createSurface // IPC Client->createSurface SurfaceFlinger->createLayer createXXXLayer() new LayerXXX Layer->getSurface() new SurfaceControl // SurfaceControl包含创建出来的ISurface setSurfaceControl // 保存到JNI Context当中
这样Isurface就创建好了
再来看另外一路发生了什么事情，Window/View System需要初始化整个Window，这样在SurfaceView当中一些callback(比如resize/new-surface/onWindowVisibilityChanged/setVisibility/onDetachedFromWindow)就会被调用到，这个时候最终会去调用updateWindow，然后IWindowSession.relayout之后就会有新的Surface被产生出来，然后通过Surface.transferFrom复制到SurfaceView的Surface当中。
还有一点注意的地方Java层的Surface(Surface.java)是如何转化为Native层的Surface(Surface.h|cpp，也就是SurfaceTextureClient)的，注意Surface.java持有一个名为mNativeSurface的Surface.h|cpp的指针，然后每次新创建Native层的Surface之后，就会把它保存到JNI Context当中，然后Java/Native就是通过这么来转换的。
接着我们就只看Native层Surface的管理，android_view_Surface.h|cpp当中有这么个方法android_view_Surface_getNativeWindow
而它又去调用一个内部方法getSurface，如下：
static sp<Surface> getSurface(JNIEnv* env, jobject surfaceObj) {
sp<Surface> result(android_view_Surface_getSurface(env, surfaceObj)); // 如果取出来为空
if (result == NULL) {
/*
* if this method is called from the WindowManager's process, it means
* the client is is not remote, and therefore is allowed to have
* a Surface (data), so we create it here.
* If we don't have a SurfaceControl, it means we're in a different
* process.
*/
SurfaceControl* const control = reinterpret_cast<SurfaceControl*>(
env->GetIntField(surfaceObj, gSurfaceClassInfo.mNativeSurfaceControl));
if (control) {
result = control->getSurface(); // 创建Surface(SurfaceTextureClient)
if (result != NULL) {
result->incStrong(surfaceObj);
env->SetIntField(surfaceObj, gSurfaceClassInfo.mNativeSurface, // Native关联变量，gui/Surface.h
reinterpret_cast<jint>(result.get()));
}
}
}
return result;
}
sp<ANativeWindow> android_view_Surface_getNativeWindow(JNIEnv* env, jobject surfaceObj) { // 这是供Native Activity使用的
return getSurface(env, surfaceObj);
}
看似这就是创建Surface的地方，实则不然，这是供Native Activity使用。我们普通的Java Activity是createFromParcel。
创建的过程当中会初始化ISurface变量，这个是从SurfaceFlinger的Layer的创建的，另外也会通过ISurface->getSurfaceTexture()取得BufferQueue，这样(Surface)SurfaceTextureClient和BufferQueue也就建立起了联系，也就能通过native_window_*或者ANativeWindow往BufferQueue里压入数据。
举个Camera的例子，我们知道在HAL当中每个Stream创建的时候都会有一个camera2_stream_ops参数传进去，并且在Stream的callback当中都会调用camera2_stream_ops->enqueue_buffer，然后调用到ANativeWindow->queueBuffer，最终会调用到BufferQueue的方法，所以你看如果我们喂给Camera HAL的ANativeWindow是SurfaceFlinger当中创建的话，那么Stream的数据就会回到SurfaceFlinger当中，SurfaceFlinger对需要的Layer的数据进行merge之后就可以给FB显示出来了，这就是Camera preview的原理。
杂项：
另外FrameBufferNativeWindow已经不再被使用了。
我们通常说在新的支持硬件加速的设备和系统上，我们倾向于使用TextureView来替代SurfaceView，这里面又是什么原因呢？
都知道SurfaceView会单独创建一个Surface，在SurfaceFlinger当中的体现也是多创建一个Layer，然后与原有的，比如Window/Status Bar等等这些Layer合并之后再在display上画出来。
那使用TextureView就不会有这么一个过程吗？是的，因为TextureView里面利用了SurfaceTexture，SurfaceTexture的创建不会导致SurfaceFlinger中多出来一个Layer，因为它是使用硬件来做的，所以TextureView必须是支持硬件加速，并且开启的情况下才能使用，否则它什么也做不了。但是它还是会创建一个Layer，只不过这个Layer是硬件来创建，管理，那软件层面就不用花这个功来做这件事情。Native层的SurfaceTexture(ConsumerBase)它负责接收过来的数据，然后通过JNI往上传View层，软件层面的工作就结束了。
P.S. 详细信息待补充
阅读资料:
JPEG File Interchange Format Version 1.02
JEITA CP-3451 Exif Version 2.2
DIGITAL COMPRESSION AND CODING OF CONTINUOUS-TONE STILL IMAGES – REQUIREMENTS AND GUIDELINES
The JPEG Still Picture Compression Standard
JPEG 简易文档 V2.15 云风
libjpeg 6b
JPEG 原理详细实例分析及其在嵌入式 Linux 中的应用 http://www.ibm.com/developerworks/cn/linux/l-cn-jpeg/
JPEG文件编/解码详解 http://blog.csdn.net/lpt19832003/article/details/1713718
Huffman 编码压缩算法 http://coolshell.cn/articles/7459.html
A SIMPLE EXAMPLE OF HUFFMAN CODING ON A STRING http://nerdaholyc.com/a-simple-example-of-huffman-coding-on-a-string/
这是一篇自己学习JPEG的笔记，会不断完善，多数是别人的啦，少许自己理解，如有理解错误还请指出！
1、JPEG File格式分为两个部分：标记码和压缩数据
标记码由2个字节组成，以0xFF打头，后面的一个字节根据含义不同而定。每个标记码之前还可以加任意个0xFF来填充，他们没有什么意义，也就是说连续的多个0xFF被解释成一个0xFF。
2、
压缩算法是JPEG
色彩空间是YCbCr
APP0标记码是必须的
字节顺都是大端
YCbCr的Cb和Cr一般都是有符号数，但是在JPEG当中都是无符号数，所以目前在JPEG当中的做法是RGB到YCbCr的转换是计算好Cb和Cr之后将其值增加128，再做后续的运算。
YCbCr到RGB的计算是先将Cb和Cr的值减去128后再来做转换。
假设一个数据单元，8 x 8的原始图像如下(转化为YCbCr之后的数据)：
52 55 61 66 70 61 64 73 63 59 55 90 109 85 69 72 62 59 68 113 144 104 66 73 63 58 71 122 154 106 70 69 67 61 68 104 126 88 68 70 79 65 60 70 77 68 58 75 85 71 64 59 55 61 65 83 87 79 69 68 65 76 78 94
3、DCT(Discrete Cosine Transform)
在做FDCT(Forward DCT)变换的时候，Y，Cb和Cr的值都范围都是-128 ~ 127，所以都要被减去128。
适配范围，每个点都减去128：
-76 -73 -67 -62 -58 -67 -64 -55 -65 -69 -73 -38 -19 -43 -59 -56 -66 -69 -60 -15 16 -24 -62 -55 -65 -70 -57 -6 26 -22 -58 -59 -61 -67 -60 -24 -2 -40 -60 -58 -49 -63 -68 -58 -51 -60 -70 -53 -43 -57 -64 -69 -73 -67 -63 -45 -41 -49 -59 -60 -63 -52 -50 -34
使用FDCT并四舍五入取最接近的整数：
-415 -30 -61 27 56 -20 -2 0 4 -22 -61 10 13 -7 -9 5 -47 7 77 -25 -29 10 5 -6 -49 12 34 -15 -10 6 2 2 12 -7 -13 -4 -2 2 -3 3 -8 3 2 -6 -2 1 4 2 -1 0 0 -2 -1 -3 4 -1 0 0 -1 -4 -1 0 1 2
DC(Direct Current)/AC(Alternating Current)
什么是DC？什么是AC？
4、量化：
-26 -3 -6 2 2 -1 0 0 0 -2 -4 1 1 0 0 0 -3 1 5 -1 -1 0 0 0 -4 1 2 -1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
5、Zigzag
Zigzag的好处，在内存当中连续的点在图片上也是相邻的了，而且后面都是连续的0，可以继续使用RLE压缩。
于是就变成：
−26,−3,0,−3,−2,−6,2,−4,1,−4,1,1,5,1,2,−1,1,−1,2,0,0,0,0,0,−1,−1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
实际上我们的DC系数在编码的时候不会算到Zigzag里面，因为DC系数值比较大，并且相邻的两个数据单元的DC差值不会很大，所以JPEG使用了差分脉冲调制编码(DPCM)对相邻两个数据单元之间的DC系数进行差值编码，这是利用了两个数据单元之间的相关性。
对其他的63个AC系数会采用Zigzag编码。
6、RLE(Run Length Coding)
我们来用一个简单的例子来详细说明一下，假设以下是使用Zigzag编码过的63个AC系数数据：
57,45,0,0,0,0,23,0,-30,-16,0,0,1,0,0,0,0,0,0,0,..,0
可以表示为
(0,57) ; (0,45) ; (4,23) ; (1,-30) ; (0,-16) ; (2，1) ; EOB
EOB实际就是结束，也可以用(0,0)表示，如果最后面不是以0结束的就不需要
解释下这个含义，编码后的含义就是说在57之前有0个0，在45之前有0个0，在23之前有4个0，在-30之前有1个0，以此类推。
需要值得注意的是，我们后面还会对这样的数据使用Huffman压缩，但是该算法要求这个表示0的个数的值的位数是4bit，也就是说能保存的值的范围是0 ~ 15。
所以碰到连续大于15个0的情况的时候，我们会拆分(15,0) ; (3,2) ; … ; (15,0) ; (15,0) ; (1,4)，用(15,0)表示连续的16个0，也就是表示：
19个0,2, … ,33个0,4。
7、Canonical Huffman Code
在做好这些工作之后，JPEG还会对数据进行压缩，利用Canonical Huffman Code编码，对出现频率更高的数据采用更短的码字来编码。
这里将数值按照位数分为了16组。
数值 组 实际保存值
0 0 -
-1,1 1 0,1
-3,-2,2,3 2 00,01,10,11
-7,-6,-5,-4,4,5,6,7 3 000,001,010,011,100,101,110,111
-15,..,-8,8,..,15 4 0000,..,0111,1000,..,1111
-31,..,-16,16,..,31 5 00000,..,01111,10000,..,11111
-63,..,-32,32,..,63 6 .
-127,..,-64,64,..,127 7 .
-255,..,-128,128,..,255 8 .
-511,..,-256,256,..,511 9 .
-1023,..,-512,512,..,1023 10 .
-2047,..,-1024,1024,..,2047 11 .
-4095,..,-2048,2048,..,4095 12 .
-8191,..,-4096,4096,..,8191 13 .
-16383,..,-8192,8192,..,16383 14 .
-32767,..,-16384,16384,..,32767 15 .
之前RLE之后的结果：
(0,57) ; (0,45) ; (4,23) ; (1,-30) ; (0,-16) ; (2，1) ; EOB
现在仅对后面的值进行变换，前面表示0的个数的值不动。
57是第6组的，实际保存值为111001，所以被编码为(6,111001)
45，同样的操作，编码为 (6,101101)
23 -> (5,10111)
-30 -> (5,00001)
-8 -> (4,0111)
1 -> (1,1)
前面的那串数字就变成了:
(0,6), 111001 ; (0,6), 101101 ; (4,5), 10111; (1,5), 00001; (0,4) , 0111 ; (2,1), 1 ; (0,0)
括号里的数值正好合成一个字节，后面被编码的数字表示范围是-32767 ~ 32767。
合成的字节里，高4位是前续0的个数，低4位描述了后面数字的位数。
接着变，下面的变化就需要去查找Huffman编码表了，这个是在压缩之前就应该构建好的，这个表是将0 ~ 255的8位定长数根据其出现的频率不同映射成为1 ~ 16位不定长数，频率大的小于8位，频率小的高于8位。这点很重要，当然这个表是如何构建出来的也很重要。
现在我们假设查表得知：
6 = (0,6) --- 111000 (注: 6 = 0 * 16 + 6 = 0x06) 69 = (4,5) --- 1111111110011001 (注: 69 = 4 * 16 + 5 = 0x45) 21 = (1,5) --- 11111110110 4 = (0,4) --- 1011 33 = (2,1) --- 11011 0 = EOB = (0,0) --- 1010
那么最终我们得到的AC系数按位流就如下：
111000 111001 111000 101101 1111111110011001 10111 11111110110 00001 1011 0111 11011 1 1010
好奇：看起来前面括号里用的编码(范围为0 ~ 255)和后面编码的数字范围为-32767 ~ 32767的来自不同的编码基础？这样不会混淆么？
8、DC编码
DC在每个数据单元之中只有一个，并且连续数据单元之间的DC有紧密的联系，所以JPEG当中就采用的差分脉冲调制编码(DPCM)。
相邻单元之间
Diff = DC(i) – DC(i-1)
所以
DC(i) = DC(i-1) + Diff
我们保存的DC都是和上一个DC的差值，所以DC(0)等于多少？云风说是0，我觉得不应该啊。
目前DC相关这部分需要进一步证实。
假设Diff为-511
就会被编码成(9, 000000000)
假设查表(一般在JPEG文件当中Y和C是不同的表，另外还分DC和AC，所以通常就有4个Huffman表)的9的Huffman编码为1111110，那么整个DC的二进制表示就为1111110 000000000。
将DC的数据流放到AC之前就可以组成完整的编码数据流。
1111110 000000000 111000 111001 111000 101101 1111111110011001 10111 11111110110 00001 1011 0111 11011 1 1010
9、疑问：
MCU和DU之间有关系吗？看起来这些压缩都是在DU当中进行的。
每个Y,Cb,Cr分量都要来这样压缩一次吗？还是3个分量一起压缩？或者说是可以根据采样的不同，将部分分量放在一起压缩，比如Cb和Cr一起压缩，Y单独压缩。
看起来Cb可以和Cr放在一起压缩。
10、JFIF文件格式分析实例：
SOI(Start of Image)
APP0(JFIF application segment)
APP1(application reserved)
…
APP15
DQT(Define Quantization Table)
DHT(Difine Huffman Table)
SOF(Start of Frame)
SOS(Start of Scan)
EOI(End of Image)
IFD(Image File Directories)
标记的前2个字节为名字，也就是前面所说的标记码，接着的2个字节为该标记的所占空间大小(不包含标记名字的2个字节，但包含自己在内)
比如这一段数据
FF E0 00 10 4A 46 49 46 00 01 01 00 00 01 00 01 00 00
当中
0xFFE0为标记码，也就是APP0
0×0010为该标记所占大小，也就是16
后面跟的14个字节(“4A 46 49 46 00 01 01 00 00 01 00 01 00 00″)就是标记内容
根据SPEC的描述：
5字节为标识符，这里就是JFIF0
2字节为版本号(主版本和次版本各占1个字节)，这里就是1.01
1字节为密度单位，0表示没有单位，1表示每英寸点数，2表示每厘米点数
2字节为水平像素密度，这里是1
2字节为垂直像素密度，这里是1
1字节为缩略图水平像素总数，这里是0
1字节为缩略图垂直像素总数，这里为0
如果没有缩略图，这两个缩略图相关的字段值都必须要为0，如果不为0就表示后面还有缩略图的RGB数据，大小是3字节的倍数。
对于DQT
FF DB 00 43 00 08 06 06 07 06 05 08 07 07 07 09 09 08 0A 0C 14 0D 0C 0B 0B 0C 19 12 13 0F 14 1D 1A 1F 1E 1D 1A 1C 1C 20 24 2E 27 20 22 2C 23 1C 1C 28 37 29 2C 30 31 34 34 34 1F 27 39 3D 38 32 3C 2E 33 34 32
FF DB 00 43 01 09 09 09 0C 0B 0C 18 0D 0D 18 32 21 1C 21 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32
0xFFDB为标记码
0×0043为所占大小，也就是67(除去本身所占2字节，大小跟精度有关)
1个字节为QT信息，高4位为QT精度，低4位为QT号
还原成矩阵形式(Zigzag)
8 6 5 8 12 20 26 31 6 6 7 10 13 29 30 28 7 7 8 12 20 29 35 28 7 9 11 15 26 44 40 31 9 11 19 28 34 55 52 39 12 18 28 32 41 52 57 46 25 32 39 44 52 61 60 51 36 46 48 49 56 50 52 50
就是这样(JPEG 原理详细实例分析及其在嵌入式 Linux 中的应用这篇文章中应该画错了)
另外一个QT也是同样的方法可以还原。
对于SOF0这段数据
FF C0 00 11 08 00 95 00 E3 03 01 22 00 02 11 01 03 11 01
当中
0xFFC0表示SOF0
2个字节为长度，0×0011，即17
1个字节为数据样本的精度，这里是8位
2个字节表示图像的高度，这里是149
2个字节表示图像的宽度，这里是227
1个字节表示颜色分量数，JPEG都是YCrCb，即3
9个字节表示颜色分量信息，这里字节数是颜色分量数 multiply 3
因为分量信息中颜色分量ID占用1个字节，水平/垂直因子占用1个字节(高4位水平，低4位垂直)，量化表占用1个字节
H 2:1:1
V 2:1:1
所以总体采样因子就是(2 * 2):(1 * 1):(1 * 1)，即4:1:1
MCU宽是水平采样因子最大值 multiply 8(记该最大值为Hmax)
MCU高是垂直采样因子最大值 multiply 8(记该最大值为Vmax)
因此这里就是(Hmax * 8):(Vmax * 8)= 16:16
如果整幅图片的高度或者宽度不是MCU的整数倍，就需要padding，解码之后丢弃大于宽度或者高度部分的数据
在数据流当中，MCU是按从左到右，从上到下来排列的。
因为每个MCU由若干数据单元组成，而数据单元又必须是8:8的，所以MCU当中数据单元的个数就是4(Hmax * Vmax)
对于DHT，如下就有4个Huffman表
FF C4 00 1F 00 00 01 05 01 01 01 01 01 01 00 00 00 00 00 00 00 00 01 02 03 04 05 06 07 08 09 0A 0B
FF C4 00 B5 10 00 02 01 03 03 02 04 03 05 05 04 04 00 00 01 7D 01 02 03 00 04 11 05 12 21 31 41 06 13 51 61 07 22 71 14 32 81 91 A1 08 23 42 B1 C1 15 52 D1 F0 24 33 62 72 82 09 0A 16 17 18 19 1A 25 26 27 28 29 2A 34 35 36 37 38 39 3A 43 44 45 46 47 48 49 4A 53 54 55 56 57 58 59 5A 63 64 65 66 67 68 69 6A 73 74 75 76 77 78 79 7A 83 84 85 86 87 88 89 8A 92 93 94 95 96 97 98 99 9A A2 A3 A4 A5 A6 A7 A8 A9 AA B2 B3 B4 B5 B6 B7 B8 B9 BA C2 C3 C4 C5 C6 C7 C8 C9 CA D2 D3 D4 D5 D6 D7 D8 D9 DA E1 E2 E3 E4 E5 E6 E7 E8 E9 EA F1 F2 F3 F4 F5 F6 F7 F8 F9 FA
FF C4 00 1F 01 00 03 01 01 01 01 01 01 01 01 01 00 00 00 00 00 00 01 02 03 04 05 06 07 08 09 0A 0B
FF C4 00 B5 11 00 02 01 02 04 04 03 04 07 05 04 04 00 01 02 77 00 01 02 03 11 04 05 21 31 06 12 41 51 07 61 71 13 22 32 81 08 14 42 91 A1 B1 C1 09 23 33 52 F0 15 62 72 D1 0A 16 24 34 E1 25 F1 17 18 19 1A 26 27 28 29 2A 35 36 37 38 39 3A 43 44 45 46 47 48 49 4A 53 54 55 56 57 58 59 5A 63 64 65 66 67 68 69 6A 73 74 75 76 77 78 79 7A 82 83 84 85 86 87 88 89 8A 92 93 94 95 96 97 98 99 9A A2 A3 A4 A5 A6 A7 A8 A9 AA B2 B3 B4 B5 B6 B7 B8 B9 BA C2 C3 C4 C5 C6 C7 C8 C9 CA D2 D3 D4 D5 D6 D7 D8 D9 DA E2 E3 E4 E5 E6 E7 E8 E9 EA F2 F3 F4 F5 F6 F7 F8 F9 FA
2个字节为标记码
2个字节为长度
1个字节为类型和ID，高4位为类型，0为DC直流，1为AC交流；低4位为ID
也就是说这里有4个Huffman编码表，直流0，交流0，直流1，交流1
16个字节为码字的数量，以第一张表为例子，没有1位的码字，2位的码字1个，3位的码字5个，4到9位的码字各1个，没有9位以上的码字，所以一共是12个码字
剩下的12字节为编码内容(这12是根据码字数量得出的，1 + 5 + 1 + 1 + 1 + 1 + 1 + 1)，
00 01 02 03 04 05 06 07 08 09 0A 0B
所以转换成二进制的话就应当如下：
位数 代码 码字 2 00 00 3 01/02/03/04/05 001/010/011/100/101 4 06 0110 5 07 00111 6 08 001000 7 09 0001001 8 0A 00001010 9 0B 000001011
这个Huffman表是如何构建的，得好好研究，另外后面的压缩都会用到这个表的内容。
EXIF通常是放在APP1当中
FF E1 66 59 …… FF D9
0xFFE1(2字节)就是APP1的Marker名字，0×6559(2字节)是该字段的长度，为26201，但是不包含TAG名字所占用的2个字节。也就是说后面的实际内容占用了26199个字节，直到0xFFD9，这就是EXIF的内容。为什么会有个0xFFD9呢？我们都知道0xFFD9是EOI，因为EXIF里面的缩略图实际上也是一个完整的JPEG图片，它也有SOI等等这一些Marker。
参见JEITA CP-3451 Exif Version 2.2的Figure 7 Structure of Exif file with compressed thumbnail
因为EXIF并不是JPEG必须的，所以你把这之间的内容整个都拿掉(用16进制编辑器很好做到，有兴趣的可以试试看)，对图片也没有什么影响，只是少了所有EXIF的信息。
11、Linux下编译libjpeg
download source tree
uncompress it
cd to source tree folder
clean the source tree
$ ./configure --enable-shared --enable-static
Maybe it will complain this message
./configure: 1562: ./ltconfig: not found // Here
but it does not matter, so we do not care it.
$ make
if it complains ‘make: ./libtool: Command not found’
then fix your configure file from
LIBTOOL=”./libtool”
to
LIBTOOL=”libtool”
‘Cause under your current source tree, there is of course no file named libtool, so it can not exec that command, so replace it with the system-wide libtool. BUT the prerequisite thing is that you have installed libtool on your host machine.
Then compile it again, everything should be okay.
After completed, remember there will be a folder name ‘.libs’ generated under your source tree folder, this is a HIDDEN folder, everything you need just under it.
$ ./cjpeg testimg.bmp > hello.jpeg
a new compressed jpeg file is generated
$ ./djpeg -bmp testimg.jpg > hello.bmp
a new bmp file is generated
use your imagination to do anything.
Good luck!
自己学习了下JPEG理论知识以后，找了个简单的解码器(NanoJPEG)试试看，原始地址在这里http://keyj.emphy.de/nanojpeg/，短短几百行，还是比较容易看懂的，本人理解详细参见代码注释，如理解有误欢迎指出，有问题/兴趣也可以留言和我讨论。
$ gcc -O3 -D_NJ_EXAMPLE_PROGRAM -o nanojpeg nanojpeg.c $ ./nanojpeg testorig.jpg
跟着main函数，边理论边实践，不错的方法！
// NanoJPEG -- KeyJ's Tiny Baseline JPEG Decoder
// version 1.3 (2012-03-05)
// by Martin J. Fiedler <martin.fiedler@gmx.net>
//
// This software is published under the terms of KeyJ's Research License,
// version 0.2. Usage of this software is subject to the following conditions:
// 0. There's no warranty whatsoever. The author(s) of this software can not
// be held liable for any damages that occur when using this software.
// 1. This software may be used freely for both non-commercial and commercial
// purposes.
// 2. This software may be redistributed freely as long as no fees are charged
// for the distribution and this license information is included.
// 3. This software may be modified freely except for this license information,
// which must not be changed in any way.
// 4. If anything other than configuration, indentation or comments have been
// altered in the code, the original author(s) must receive a copy of the
// modified code.
///////////////////////////////////////////////////////////////////////////////
// DOCUMENTATION SECTION //
// read this if you want to know what this is all about //
///////////////////////////////////////////////////////////////////////////////
// INTRODUCTION
// ============
//
// This is a minimal decoder for baseline JPEG images. It accepts memory dumps
// of JPEG files as input and generates either 8-bit grayscale or packed 24-bit
// RGB images as output. It does not parse JFIF or Exif headers; all JPEG files
// are assumed to be either grayscale or YCbCr. CMYK or other color spaces are
// not supported. All YCbCr subsampling schemes with power-of-two ratios are
// supported, as are restart intervals. Progressive or lossless JPEG is not
// supported.
// Summed up, NanoJPEG should be able to decode all images from digital cameras
// and most common forms of other non-progressive JPEG images.
// The decoder is not optimized for speed, it's optimized for simplicity and
// small code. Image quality should be at a reasonable level. A bicubic chroma
// upsampling filter ensures that subsampled YCbCr images are rendered in
// decent quality. The decoder is not meant to deal with broken JPEG files in
// a graceful manner; if anything is wrong with the bitstream, decoding will
// simply fail.
// The code should work with every modern C compiler without problems and
// should not emit any warnings. It uses only (at least) 32-bit integer
// arithmetic and is supposed to be endianness independent and 64-bit clean.
// However, it is not thread-safe.
// COMPILE-TIME CONFIGURATION
// ==========================
//
// The following aspects of NanoJPEG can be controlled with preprocessor
// defines:
//
// _NJ_EXAMPLE_PROGRAM = Compile a main() function with an example
// program.
// _NJ_INCLUDE_HEADER_ONLY = Don't compile anything, just act as a header
// file for NanoJPEG. Example:
// #define _NJ_INCLUDE_HEADER_ONLY
// #include "nanojpeg.c"
// int main(void) {
// njInit();
// // your code here
// njDone();
// }
// NJ_USE_LIBC=1 = Use the malloc(), free(), memset() and memcpy()
// functions from the standard C library (default).
// NJ_USE_LIBC=0 = Don't use the standard C library. In this mode,
// external functions njAlloc(), njFreeMem(),
// njFillMem() and njCopyMem() need to be defined
// and implemented somewhere.
// NJ_USE_WIN32=0 = Normal mode (default).
// NJ_USE_WIN32=1 = If compiling with MSVC for Win32 and
// NJ_USE_LIBC=0, NanoJPEG will use its own
// implementations of the required C library
// functions (default if compiling with MSVC and
// NJ_USE_LIBC=0).
// NJ_CHROMA_FILTER=1 = Use the bicubic chroma upsampling filter
// (default). // 图像resize的一种算法
// NJ_CHROMA_FILTER=0 = Use simple pixel repetition for chroma upsampling
// (bad quality, but faster and less code).
// API
// ===
//
// For API documentation, read the "header section" below.
// EXAMPLE
// =======
//
// A few pages below, you can find an example program that uses NanoJPEG to
// convert JPEG files into PGM or PPM. To compile it, use something like
// gcc -O3 -D_NJ_EXAMPLE_PROGRAM -o nanojpeg nanojpeg.c
// You may also add -std=c99 -Wall -Wextra -pedantic -Werror, if you want :)
///////////////////////////////////////////////////////////////////////////////
// HEADER SECTION //
// copy and pase this into nanojpeg.h if you want //
///////////////////////////////////////////////////////////////////////////////
#ifndef _NANOJPEG_H
#define _NANOJPEG_H
// nj_result_t: Result codes for njDecode().
typedef enum _nj_result {
NJ_OK = 0, // no error, decoding successful
NJ_NO_JPEG, // not a JPEG file
NJ_UNSUPPORTED, // unsupported format
NJ_OUT_OF_MEM, // out of memory
NJ_INTERNAL_ERR, // internal error
NJ_SYNTAX_ERROR, // syntax error
__NJ_FINISHED, // used internally, will never be reported
} nj_result_t;
// njInit: Initialize NanoJPEG.
// For safety reasons, this should be called at least one time before using
// using any of the other NanoJPEG functions.
void njInit(void);
// njDecode: Decode a JPEG image.
// Decodes a memory dump of a JPEG file into internal buffers.
// Parameters:
// jpeg = The pointer to the memory dump.
// size = The size of the JPEG file.
// Return value: The error code in case of failure, or NJ_OK (zero) on success.
nj_result_t njDecode(const void* jpeg, const int size);
// njGetWidth: Return the width (in pixels) of the most recently decoded
// image. If njDecode() failed, the result of njGetWidth() is undefined.
int njGetWidth(void);
// njGetHeight: Return the height (in pixels) of the most recently decoded
// image. If njDecode() failed, the result of njGetHeight() is undefined.
int njGetHeight(void);
// njIsColor: Return 1 if the most recently decoded image is a color image
// (RGB) or 0 if it is a grayscale image. If njDecode() failed, the result
// of njGetWidth() is undefined.
int njIsColor(void);
// njGetImage: Returns the decoded image data.
// Returns a pointer to the most recently image. The memory layout it byte-
// oriented, top-down, without any padding between lines. Pixels of color
// images will be stored as three consecutive bytes for the red, green and
// blue channels. This data format is thus compatible with the PGM or PPM
// file formats and the OpenGL texture formats GL_LUMINANCE8 or GL_RGB8.
// If njDecode() failed, the result of njGetImage() is undefined.
unsigned char* njGetImage(void);
// njGetImageSize: Returns the size (in bytes) of the image data returned
// by njGetImage(). If njDecode() failed, the result of njGetImageSize() is
// undefined.
int njGetImageSize(void);
// njDone: Uninitialize NanoJPEG.
// Resets NanoJPEG's internal state and frees all memory that has been
// allocated at run-time by NanoJPEG. It is still possible to decode another
// image after a njDone() call.
void njDone(void);
#endif//_NANOJPEG_H
///////////////////////////////////////////////////////////////////////////////
// CONFIGURATION SECTION //
// adjust the default settings for the NJ_ defines here //
///////////////////////////////////////////////////////////////////////////////
#ifndef NJ_USE_LIBC
#define NJ_USE_LIBC 1
#endif
#ifndef NJ_USE_WIN32
#ifdef _MSC_VER
#define NJ_USE_WIN32 (!NJ_USE_LIBC)
#else
#define NJ_USE_WIN32 0
#endif
#endif
#ifndef NJ_CHROMA_FILTER
#define NJ_CHROMA_FILTER 1
#endif
///////////////////////////////////////////////////////////////////////////////
// EXAMPLE PROGRAM //
// just define _NJ_EXAMPLE_PROGRAM to compile this (requires NJ_USE_LIBC) //
///////////////////////////////////////////////////////////////////////////////
#ifdef _NJ_EXAMPLE_PROGRAM
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(int argc, char* argv[]) {
int size;
char *buf;
FILE *f;
if (argc < 2) {
printf("Usage: %s <input.jpg> [<output.ppm>]\n", argv[0]);
return 2;
}
f = fopen(argv[1], "rb");
if (!f) {
printf("Error opening the input file.\n");
return 1;
}
fseek(f, 0, SEEK_END);
size = (int) ftell(f); // 字节
buf = malloc(size);
fseek(f, 0, SEEK_SET);
size = (int) fread(buf, 1, size, f); // 读取整个文件内容到buf
fclose(f);
njInit(); // 初始化nj_context_t
if (njDecode(buf, size)) {
printf("Error decoding the input file.\n");
return 1;
}
f = fopen((argc > 2) ? argv[2] : (njIsColor() ? "nanojpeg_out.ppm" : "nanojpeg_out.pgm"), "wb");
if (!f) {
printf("Error opening the output file.\n");
return 1;
}
fprintf(f, "P%d\n%d %d\n255\n", njIsColor() ? 6 : 5, njGetWidth(), njGetHeight());
fwrite(njGetImage(), 1, njGetImageSize(), f);
fclose(f);
njDone();
return 0;
}
#endif
// 解释什么是stride http://msdn.microsoft.com/en-us/library/windows/desktop/aa473780(v=vs.85).aspx
///////////////////////////////////////////////////////////////////////////////
// IMPLEMENTATION SECTION //
// you may stop reading here //
///////////////////////////////////////////////////////////////////////////////
#ifndef _NJ_INCLUDE_HEADER_ONLY
#ifdef _MSC_VER
#define NJ_INLINE static __inline
#define NJ_FORCE_INLINE static __forceinline
#else
#define NJ_INLINE static inline
#define NJ_FORCE_INLINE static inline
#endif
#if NJ_USE_LIBC
#include <stdlib.h>
#include <string.h>
#define njAllocMem malloc
#define njFreeMem free
#define njFillMem memset
#define njCopyMem memcpy
#elif NJ_USE_WIN32
#include <windows.h>
#define njAllocMem(size) ((void*) LocalAlloc(LMEM_FIXED, (SIZE_T)(size)))
#define njFreeMem(block) ((void) LocalFree((HLOCAL) block))
NJ_INLINE void njFillMem(void* block, unsigned char value, int count) { __asm {
mov edi, block
mov al, value
mov ecx, count
rep stosb
} }
NJ_INLINE void njCopyMem(void* dest, const void* src, int count) { __asm {
mov edi, dest
mov esi, src
mov ecx, count
rep movsb
} }
#else
extern void* njAllocMem(int size);
extern void njFreeMem(void* block);
extern void njFillMem(void* block, unsigned char byte, int size);
extern void njCopyMem(void* dest, const void* src, int size);
#endif
typedef struct _nj_code {
unsigned char bits, code;
} nj_vlc_code_t;
typedef struct _nj_cmp {
int cid;
int ssx, ssy; // 水平/垂直因子
int width, height;
int stride;
int qtsel; // Quantization Table量化表
int actabsel, dctabsel; // AC/DC Huffman Table
int dcpred;
unsigned char *pixels;
} nj_component_t; // 颜色分量
typedef struct _nj_ctx {
nj_result_t error;
const unsigned char *pos; // 待解码数据指针(按字节来)
int size; // 整个数据的长度
int length; // 某一个marker内容的长度
int width, height; // 图片宽和高度
int mbwidth, mbheight; // MCU水平/垂直个数
int mbsizex, mbsizey; // MCU宽/高
int ncomp; // 颜色分量数
nj_component_t comp[3]; // YCbCr
int qtused, qtavail; // 这两个目前看不出来很大用处
unsigned char qtab[4][64]; // 但是目前似乎只有2个
nj_vlc_code_t vlctab[4][65536]; // 构造所有16位数的Huffman基数
// 目前基本上是4个(直/交/0/1)
int buf, bufbits; // 这是用来做什么的 buf是存放内容的 bufbits是计数器，存放了多少个bits
int block[64];
int rstinterval;
unsigned char *rgb; // 解析出来的RGB所要占用的内存 // 每1个点包含3个字节，按找RGB的顺序
} nj_context_t;
static nj_context_t nj;
static const char njZZ[64] = { 0, 1, 8, 16, 9, 2, 3, 10, 17, 24, 32, 25, 18,
11, 4, 5, 12, 19, 26, 33, 40, 48, 41, 34, 27, 20, 13, 6, 7, 14, 21, 28, 35,
42, 49, 56, 57, 50, 43, 36, 29, 22, 15, 23, 30, 37, 44, 51, 58, 59, 52, 45,
38, 31, 39, 46, 53, 60, 61, 54, 47, 55, 62, 63 };
/*
0 1 2 3 4 5 6 7
8 9 10 11 12 13 14 15
16 17 18 19 20 21 22 23
24 25 26 27 28 29 30 31
32 33 34 35 36 37 38 39
40 41 42 43 44 45 46 47
48 49 50 51 52 53 54 55
56 57 58 59 60 61 62 63
*/
NJ_FORCE_INLINE unsigned char njClip(const int x) { // 限定范围是0 ~ 255之间
return (x < 0) ? 0 : ((x > 0xFF) ? 0xFF : (unsigned char) x);
}
#define W1 2841
#define W2 2676
#define W3 2408
#define W5 1609
#define W6 1108
#define W7 565
NJ_INLINE void njRowIDCT(int* blk) { // 按行来操作的 0 ~ 7 // 8 ~ 15
int x0, x1, x2, x3, x4, x5, x6, x7, x8;
if (!((x1 = blk[4] << 11)
| (x2 = blk[6])
| (x3 = blk[2])
| (x4 = blk[1])
| (x5 = blk[7])
| (x6 = blk[5])
| (x7 = blk[3])))
{
blk[0] = blk[1] = blk[2] = blk[3] = blk[4] = blk[5] = blk[6] = blk[7] = blk[0] << 3;
return;
}
x0 = (blk[0] << 11) + 128;
x8 = W7 * (x4 + x5);
x4 = x8 + (W1 - W7) * x4;
x5 = x8 - (W1 + W7) * x5;
x8 = W3 * (x6 + x7);
x6 = x8 - (W3 - W5) * x6;
x7 = x8 - (W3 + W5) * x7;
x8 = x0 + x1;
x0 -= x1;
x1 = W6 * (x3 + x2);
x2 = x1 - (W2 + W6) * x2;
x3 = x1 + (W2 - W6) * x3;
x1 = x4 + x6;
x4 -= x6;
x6 = x5 + x7;
x5 -= x7;
x7 = x8 + x3;
x8 -= x3;
x3 = x0 + x2;
x0 -= x2;
x2 = (181 * (x4 + x5) + 128) >> 8;
x4 = (181 * (x4 - x5) + 128) >> 8;
blk[0] = (x7 + x1) >> 8;
blk[1] = (x3 + x2) >> 8;
blk[2] = (x0 + x4) >> 8;
blk[3] = (x8 + x6) >> 8;
blk[4] = (x8 - x6) >> 8;
blk[5] = (x0 - x4) >> 8;
blk[6] = (x3 - x2) >> 8;
blk[7] = (x7 - x1) >> 8;
}
NJ_INLINE void njColIDCT(const int* blk, unsigned char *out, int stride) {
int x0, x1, x2, x3, x4, x5, x6, x7, x8;
if (!((x1 = blk[8*4] << 8)
| (x2 = blk[8*6])
| (x3 = blk[8*2])
| (x4 = blk[8*1])
| (x5 = blk[8*7])
| (x6 = blk[8*5])
| (x7 = blk[8*3])))
{
x1 = njClip(((blk[0] + 32) >> 6) + 128);
for (x0 = 8; x0; --x0) {
*out = (unsigned char) x1;
out += stride;
}
return;
}
x0 = (blk[0] << 8) + 8192;
x8 = W7 * (x4 + x5) + 4;
x4 = (x8 + (W1 - W7) * x4) >> 3;
x5 = (x8 - (W1 + W7) * x5) >> 3;
x8 = W3 * (x6 + x7) + 4;
x6 = (x8 - (W3 - W5) * x6) >> 3;
x7 = (x8 - (W3 + W5) * x7) >> 3;
x8 = x0 + x1;
x0 -= x1;
x1 = W6 * (x3 + x2) + 4;
x2 = (x1 - (W2 + W6) * x2) >> 3;
x3 = (x1 + (W2 - W6) * x3) >> 3;
x1 = x4 + x6;
x4 -= x6;
x6 = x5 + x7;
x5 -= x7;
x7 = x8 + x3;
x8 -= x3;
x3 = x0 + x2;
x0 -= x2;
x2 = (181 * (x4 + x5) + 128) >> 8; // Y，Cb和Cr的值都范围都是-128 ~ 127，并且在FDCT的时候有先减去128，所以现在要IDCT之后再加上128
x4 = (181 * (x4 - x5) + 128) >> 8;
*out = njClip(((x7 + x1) >> 14) + 128); out += stride;
*out = njClip(((x3 + x2) >> 14) + 128); out += stride;
*out = njClip(((x0 + x4) >> 14) + 128); out += stride;
*out = njClip(((x8 + x6) >> 14) + 128); out += stride;
*out = njClip(((x8 - x6) >> 14) + 128); out += stride;
*out = njClip(((x0 - x4) >> 14) + 128); out += stride;
*out = njClip(((x3 - x2) >> 14) + 128); out += stride;
*out = njClip(((x7 - x1) >> 14) + 128);
}
#define njThrow(e) do { nj.error = e; return; } while (0)
#define njCheckError() do { if (nj.error) return; } while (0)
static int njShowBits(int bits) { // 能放得下大于32位的值么？
unsigned char newbyte;
if (!bits) return 0;
while (nj.bufbits < bits) { // 也就是说要buf的位数小于已经buf的位数的时候，就直接读出来？
if (nj.size <= 0) {
nj.buf = (nj.buf << 8) | 0xFF;
nj.bufbits += 8;
continue;
}
newbyte = *nj.pos++; // 数据指针是按字节
nj.size--;
nj.bufbits += 8;
nj.buf = (nj.buf << 8) | newbyte; // 高位最终会被覆盖掉，比如我要buf一个64位的值怎么办？
if (newbyte == 0xFF) {
if (nj.size) {
unsigned char marker = *nj.pos++;
nj.size--;
switch (marker) {
case 0x00:
case 0xFF:
break;
case 0xD9: nj.size = 0; break;
default:
if ((marker & 0xF8) != 0xD0)
nj.error = NJ_SYNTAX_ERROR;
else {
nj.buf = (nj.buf << 8) | marker;
nj.bufbits += 8;
}
}
} else
nj.error = NJ_SYNTAX_ERROR;
}
}
return (nj.buf >> (nj.bufbits - bits)) & ((1 << bits) - 1);
}
NJ_INLINE void njSkipBits(int bits) {
if (nj.bufbits < bits)
(void) njShowBits(bits);
nj.bufbits -= bits;
}
NJ_INLINE int njGetBits(int bits) {
int res = njShowBits(bits);
njSkipBits(bits);
return res;
}
NJ_INLINE void njByteAlign(void) {
nj.bufbits &= 0xF8; // (1111 1000)8的倍数，不满8的部分丢弃
}
static void njSkip(int count) {
nj.pos += count; // 数据指针增加
nj.size -= count; // 总体数据大小减去count
nj.length -= count; // 当前marker长度减去count
if (nj.size < 0) nj.error = NJ_SYNTAX_ERROR;
}
NJ_INLINE unsigned short njDecode16(const unsigned char *pos) {
return (pos[0] << 8) | pos[1]; // 00000000 00001101
}
static void njDecodeLength(void) { // decode长度字段，这个方法调用一般都是已经进入到特定的marker之后
if (nj.size < 2) njThrow(NJ_SYNTAX_ERROR);
nj.length = njDecode16(nj.pos); // 该marker的长度(除去marker名字所占用的2个字节)
if (nj.length > nj.size) njThrow(NJ_SYNTAX_ERROR);
njSkip(2);
}
NJ_INLINE void njSkipMarker(void) {
njDecodeLength();
njSkip(nj.length);
}
NJ_INLINE void njDecodeSOF(void) { // 解析Start of Frame的时候就会把所需要的内存都分配好
int i, ssxmax = 0, ssymax = 0;
nj_component_t* c;
njDecodeLength(); // 解析长度并移动数据指针
if (nj.length < 9) njThrow(NJ_SYNTAX_ERROR);
if (nj.pos[0] != 8) njThrow(NJ_UNSUPPORTED); // 样本精度，一般都是8
nj.height = njDecode16(nj.pos + 1); // 图片高度/宽度
nj.width = njDecode16(nj.pos + 3);
nj.ncomp = nj.pos[5]; // 颜色分量数据，一般都是3
njSkip(6); // 之前共6个字节数据，所以移动数据指针6个字节
switch (nj.ncomp) { // 目前只支持1和3这两种
case 1:
case 3:
break;
default:
njThrow(NJ_UNSUPPORTED);
}
if (nj.length < (nj.ncomp * 3)) njThrow(NJ_SYNTAX_ERROR); // 数据量肯定是要大于颜色分量数 multiply 3，因为接着存颜色分量信息的每个结构占3个字节
// 颜色分量ID占用1个字节，水平/垂直因子占用1个字节(高4位水平，低4位垂直)，量化表占用1个字节
for (i = 0, c = nj.comp; i < nj.ncomp; ++i, ++c) {
c->cid = nj.pos[0]; // 颜色分量ID
if (!(c->ssx = nj.pos[1] >> 4)) njThrow(NJ_SYNTAX_ERROR); // 高4位(水平因子)
if (c->ssx & (c->ssx - 1)) njThrow(NJ_UNSUPPORTED); // non-power of two
if (!(c->ssy = nj.pos[1] & 15)) njThrow(NJ_SYNTAX_ERROR); // (00001111)低4位(垂直因子)
if (c->ssy & (c->ssy - 1)) njThrow(NJ_UNSUPPORTED); // non-power of two
if ((c->qtsel = nj.pos[2]) & 0xFC) njThrow(NJ_SYNTAX_ERROR); // (11111101) 这里0xFC是用在这里干什么的？
njSkip(3); // 移动数据指针到下一个颜色分量
nj.qtused |= 1 << c->qtsel; // 这里是做什么用的？看不出来
if (c->ssx > ssxmax) ssxmax = c->ssx; // 记录最大水平因子
if (c->ssy > ssymax) ssymax = c->ssy; // 记录最大垂直因子
}
if (nj.ncomp == 1) { // 只有一种颜色分量的时候就简单啦
c = nj.comp;
c->ssx = c->ssy = ssxmax = ssymax = 1;
}
nj.mbsizex = ssxmax << 3; // MCU宽 是 水平采样因子最大值 multiply 8
nj.mbsizey = ssymax << 3; // MCU高 是 垂直采样因子最大值 multiply 8
nj.mbwidth = (nj.width + nj.mbsizex - 1) / nj.mbsizex; // 分子采用+ nj.mbsizex - 1就取到大于但是最接近(等于)宽度的值，
// 并且这个值是MCU宽度整数倍 // 这里是水平方向MCU的个数
nj.mbheight = (nj.height + nj.mbsizey - 1) / nj.mbsizey; // 这里是垂直方向MCU的个数
for (i = 0, c = nj.comp; i < nj.ncomp; ++i, ++c) {
c->width = (nj.width * c->ssx + ssxmax - 1) / ssxmax; // 采样宽度？ 最大水平/垂直因子的值就是图片原来的值，否则就会根据因子做相应的减少
c->stride = (c->width + 7) & 0x7FFFFFF8; // (0111 1111 1111 1111 1111 1111 1111 1000) 做什么？以1234567结尾的都省略掉？
// 变成8的整数
// 补齐8位，注意前面有加7，所以总是不会比原来的少，比如原来是227，那么这里就会变成232
// 这是按照数据单元计算的，所以不对
printf("%d, stride %d\n", i, c->stride);
c->height = (nj.height * c->ssy + ssymax - 1) / ssymax;
c->stride = nj.mbwidth * nj.mbsizex * c->ssx / ssxmax; // 再计算一遍stride有什么用？前面计算的是错误的，没有考虑MCU宽度
// 这里都已经是round过的了，所以直接计算
printf("%d, stride again %d\n", i, c->stride);
if (((c->width < 3) && (c->ssx != ssxmax)) || ((c->height < 3) && (c->ssy != ssymax))) njThrow(NJ_UNSUPPORTED);
if (!(c->pixels = njAllocMem(c->stride * (nj.mbheight * nj.mbsizey * c->ssy / ssymax)))) njThrow(NJ_OUT_OF_MEM); // 为分量分配内存
// 大小是所有MCU的
// 可能比图片实际
// 尺寸大
}
if (nj.ncomp == 3) { // 只有有3个颜色分量的时候才需要
nj.rgb = njAllocMem(nj.width * nj.height * nj.ncomp);
if (!nj.rgb) njThrow(NJ_OUT_OF_MEM);
}
njSkip(nj.length);
}
NJ_INLINE void njDecodeDHT(void) {
int codelen, currcnt, remain, spread, i, j;
nj_vlc_code_t *vlc;
static unsigned char counts[16]; // 码字
njDecodeLength();
while (nj.length >= 17) { // 码字的数量(16) + 类型和ID(1)
i = nj.pos[0]; // 类型和ID
if (i & 0xEC) njThrow(NJ_SYNTAX_ERROR); // (11101100)
if (i & 0x02) njThrow(NJ_UNSUPPORTED); // (00000010)
i = (i | (i >> 3)) & 3; // combined DC/AC + tableid value
// 直流0，直流1，交流0，交流1
for (codelen = 1; codelen <= 16; ++codelen) // 码字长度
counts[codelen - 1] = nj.pos[codelen]; // 读取码字
njSkip(17);
vlc = &nj.vlctab[i][0];
remain = spread = 65536;
for (codelen = 1; codelen <= 16; ++codelen) {
spread >>= 1; // 干什么？
currcnt = counts[codelen - 1];
if (!currcnt) continue; // 如果该位数没有码字
if (nj.length < currcnt) njThrow(NJ_SYNTAX_ERROR);
remain -= currcnt << (16 - codelen);
if (remain < 0) njThrow(NJ_SYNTAX_ERROR);
for (i = 0; i < currcnt; ++i) { // 码字个数，同样位数的码字可以有多个
register unsigned char code = nj.pos[i];
for (j = spread; j; --j) { // 保存这么多个有什么作用？
vlc->bits = (unsigned char) codelen; // 码字位数
vlc->code = code; // 码字值
++vlc;
}
}
njSkip(currcnt);
}
while (remain--) {
vlc->bits = 0;
++vlc;
}
}
if (nj.length) njThrow(NJ_SYNTAX_ERROR);
}
NJ_INLINE void njDecodeDQT(void) {
int i;
unsigned char *t;
njDecodeLength();
while (nj.length >= 65) {
i = nj.pos[0]; // QT信息，高4位为QT精度，低4位为QT号
if (i & 0xFC) njThrow(NJ_SYNTAX_ERROR); // (1111 1110)这个用来检测QT号码是否正确的吗？目前精度好像都为0，所以这么写？
nj.qtavail |= 1 << i; // XXX 直接通过这里转换为数量？
t = &nj.qtab[i][0];
for (i = 0; i < 64; ++i)
t[i] = nj.pos[i + 1]; // 读取到QT数组当中，但应该还是按照文件流当中的排列
njSkip(65);
}
if (nj.length) njThrow(NJ_SYNTAX_ERROR);
}
NJ_INLINE void njDecodeDRI(void) {
njDecodeLength();
if (nj.length < 2) njThrow(NJ_SYNTAX_ERROR);
nj.rstinterval = njDecode16(nj.pos);
njSkip(nj.length);
}
static int njGetVLC(nj_vlc_code_t* vlc, unsigned char* code) { // Variable Length Coding
int value = njShowBits(16);
int bits = vlc[value].bits;
if (!bits) { nj.error = NJ_SYNTAX_ERROR; return 0; }
njSkipBits(bits);
value = vlc[value].code;
if (code) *code = (unsigned char) value;
bits = value & 15;
if (!bits) return 0;
value = njGetBits(bits);
if (value < (1 << (bits - 1)))
value += ((-1) << bits) + 1;
return value;
}
NJ_INLINE void njDecodeBlock(nj_component_t* c, unsigned char* out) {
unsigned char code = 0;
int value, coef = 0;
njFillMem(nj.block, 0, sizeof(nj.block));
c->dcpred += njGetVLC(&nj.vlctab[c->dctabsel][0], NULL); // DC 0/1 不会和AC重复
nj.block[0] = (c->dcpred) * nj.qtab[c->qtsel][0]; // DC // 这里是反量化？
do {
value = njGetVLC(&nj.vlctab[c->actabsel][0], &code); // DC 2/3
if (!code) break; // EOB
if (!(code & 0x0F) && (code != 0xF0)) njThrow(NJ_SYNTAX_ERROR);
coef += (code >> 4) + 1; // coefficient 系数
if (coef > 63) njThrow(NJ_SYNTAX_ERROR);
nj.block[(int) njZZ[coef]] = value * nj.qtab[c->qtsel][coef]; // AC 这里是反量化？
} while (coef < 63);
for (coef = 0; coef < 64; coef += 8)
njRowIDCT(&nj.block[coef]); // 上面先Huffman解码/反量化，这里行(反DCT)
for (coef = 0; coef < 8; ++coef)
njColIDCT(&nj.block[coef], &out[coef], c->stride);
}
NJ_INLINE void njDecodeScan(void) {
int i, mbx, mby, sbx, sby;
int rstcount = nj.rstinterval, nextrst = 0;
nj_component_t* c;
njDecodeLength();
if (nj.length < (4 + 2 * nj.ncomp)) njThrow(NJ_SYNTAX_ERROR);
if (nj.pos[0] != nj.ncomp) njThrow(NJ_UNSUPPORTED);
njSkip(1); // 颜色分量数量
for (i = 0, c = nj.comp; i < nj.ncomp; ++i, ++c) {
if (nj.pos[0] != c->cid) njThrow(NJ_SYNTAX_ERROR); // 颜色分量ID
if (nj.pos[1] & 0xEE) njThrow(NJ_SYNTAX_ERROR);
c->dctabsel = nj.pos[1] >> 4; // 高4位为直流表DC Table
c->actabsel = (nj.pos[1] & 1) | 2; // 低4位为交流表AC Table(这里有做特殊处理，所以AC的表名不会和DC相同)
printf("DC/AC Huffman table ids: %d/%d\n", c->dctabsel, c->actabsel);
njSkip(2);
}
if (nj.pos[0] || (nj.pos[1] != 63) || nj.pos[2]) njThrow(NJ_UNSUPPORTED);
njSkip(nj.length); // 忽略3个字节 通常为 00 3F 00
// 2 + 1 + 6 + 3为12字节，这个marker的长度刚好为12字节
// 接下来都是编码过的图像数据
for (mbx = mby = 0;;) {
for (i = 0, c = nj.comp; i < nj.ncomp; ++i, ++c) // 每个分量都要decode
for (sby = 0; sby < c->ssy; ++sby) // 水平/垂直因子
for (sbx = 0; sbx < c->ssx; ++sbx) {
njDecodeBlock(c, &c->pixels[((mby * c->ssy + sby) * c->stride + mbx * c->ssx + sbx) << 3]); // 读取原始编码过
// 的图片数据到block中
// 并反量化，反离散余弦变换
njCheckError();
}
if (++mbx >= nj.mbwidth) { // 读完所有的MCU，到达最右就返回从下一行开始
mbx = 0;
if (++mby >= nj.mbheight) break; // 到达最底行的时候推出，decode结束
}
if (nj.rstinterval && !(--rstcount)) { // restart marker
njByteAlign();
i = njGetBits(16);
if (((i & 0xFFF8) != 0xFFD0) || ((i & 7) != nextrst)) njThrow(NJ_SYNTAX_ERROR);
nextrst = (nextrst + 1) & 7;
rstcount = nj.rstinterval;
for (i = 0; i < 3; ++i)
nj.comp[i].dcpred = 0;
}
}
nj.error = __NJ_FINISHED;
}
#if NJ_CHROMA_FILTER
#define CF4A (-9)
#define CF4B (111)
#define CF4C (29)
#define CF4D (-3)
#define CF3A (28)
#define CF3B (109)
#define CF3C (-9)
#define CF3X (104)
#define CF3Y (27)
#define CF3Z (-3)
#define CF2A (139)
#define CF2B (-11)
#define CF(x) njClip(((x) + 64) >> 7)
// 通常我们放大图片的时候就需要upsampling，缩小的时候就downsampling，通称为resampling
// 这里Cb/Cr分量的会少些，所以需要upsampling
NJ_INLINE void njUpsampleH(nj_component_t* c) {
printf("njUpsampleH %d\n", c->cid);
const int xmax = c->width - 3;
unsigned char *out, *lin, *lout;
int x, y;
out = njAllocMem((c->width * c->height) << 1);
if (!out) njThrow(NJ_OUT_OF_MEM);
lin = c->pixels;
lout = out;
for (y = c->height; y; --y) {
lout[0] = CF(CF2A * lin[0] + CF2B * lin[1]);
lout[1] = CF(CF3X * lin[0] + CF3Y * lin[1] + CF3Z * lin[2]);
lout[2] = CF(CF3A * lin[0] + CF3B * lin[1] + CF3C * lin[2]);
for (x = 0; x < xmax; ++x) {
lout[(x << 1) + 3] = CF(CF4A * lin[x] + CF4B * lin[x + 1] + CF4C * lin[x + 2] + CF4D * lin[x + 3]);
lout[(x << 1) + 4] = CF(CF4D * lin[x] + CF4C * lin[x + 1] + CF4B * lin[x + 2] + CF4A * lin[x + 3]);
}
lin += c->stride;
lout += c->width << 1;
lout[-3] = CF(CF3A * lin[-1] + CF3B * lin[-2] + CF3C * lin[-3]);
lout[-2] = CF(CF3X * lin[-1] + CF3Y * lin[-2] + CF3Z * lin[-3]);
lout[-1] = CF(CF2A * lin[-1] + CF2B * lin[-2]);
}
c->width <<= 1;
c->stride = c->width;
njFreeMem(c->pixels);
c->pixels = out;
}
NJ_INLINE void njUpsampleV(nj_component_t* c) {
printf("njUpsampleV %d\n", c->cid);
const int w = c->width, s1 = c->stride, s2 = s1 + s1;
unsigned char *out, *cin, *cout;
int x, y;
out = njAllocMem((c->width * c->height) << 1);
if (!out) njThrow(NJ_OUT_OF_MEM);
for (x = 0; x < w; ++x) {
cin = &c->pixels[x];
cout = &out[x];
*cout = CF(CF2A * cin[0] + CF2B * cin[s1]); cout += w;
*cout = CF(CF3X * cin[0] + CF3Y * cin[s1] + CF3Z * cin[s2]); cout += w;
*cout = CF(CF3A * cin[0] + CF3B * cin[s1] + CF3C * cin[s2]); cout += w;
cin += s1;
for (y = c->height - 3; y; --y) {
*cout = CF(CF4A * cin[-s1] + CF4B * cin[0] + CF4C * cin[s1] + CF4D * cin[s2]); cout += w;
*cout = CF(CF4D * cin[-s1] + CF4C * cin[0] + CF4B * cin[s1] + CF4A * cin[s2]); cout += w;
cin += s1;
}
cin += s1;
*cout = CF(CF3A * cin[0] + CF3B * cin[-s1] + CF3C * cin[-s2]); cout += w;
*cout = CF(CF3X * cin[0] + CF3Y * cin[-s1] + CF3Z * cin[-s2]); cout += w;
*cout = CF(CF2A * cin[0] + CF2B * cin[-s1]);
}
c->height <<= 1;
c->stride = c->width;
njFreeMem(c->pixels);
c->pixels = out;
}
#else
NJ_INLINE void njUpsample(nj_component_t* c) {
printf("njUpsample %d\n", c->cid);
int x, y, xshift = 0, yshift = 0;
unsigned char *out, *lin, *lout;
while (c->width < nj.width) { c->width <<= 1; ++xshift; }
while (c->height < nj.height) { c->height <<= 1; ++yshift; }
out = njAllocMem(c->width * c->height); // 放大后的尺寸
if (!out) njThrow(NJ_OUT_OF_MEM);
lin = c->pixels;
lout = out;
for (y = 0; y < c->height; ++y) {
lin = &c->pixels[(y >> yshift) * c->stride];
for (x = 0; x < c->width; ++x)
lout[x] = lin[x >> xshift];
lout += c->width;
}
c->stride = c->width;
njFreeMem(c->pixels);
c->pixels = out;
}
#endif
NJ_INLINE void njConvert() {
int i;
nj_component_t* c;
for (i = 0, c = nj.comp; i < nj.ncomp; ++i, ++c) { // 如果需要的话就upsampling
#if NJ_CHROMA_FILTER
while ((c->width < nj.width) || (c->height < nj.height)) {
if (c->width < nj.width) njUpsampleH(c);
njCheckError();
if (c->height < nj.height) njUpsampleV(c);
njCheckError();
}
#else
if ((c->width < nj.width) || (c->height < nj.height))
njUpsample(c);
#endif
if ((c->width < nj.width) || (c->height < nj.height)) njThrow(NJ_INTERNAL_ERR);
}
if (nj.ncomp == 3) { // SEE njGetImage()
// convert to RGB
int x, yy;
unsigned char *prgb = nj.rgb;
const unsigned char *py = nj.comp[0].pixels;
const unsigned char *pcb = nj.comp[1].pixels;
const unsigned char *pcr = nj.comp[2].pixels;
// 多余的数据(编/解码是对齐用的)会被丢弃吗？
for (yy = nj.height; yy; --yy) { // 列
for (x = 0; x < nj.width; ++x) { // 行
register int y = py[x] << 8; // 这是为什么？ 色彩空间转换公式计算需要
register int cb = pcb[x] - 128; // YCbCr的Cb和Cr一般都是有符号数，但是在JPEG当中都是无符号数
register int cr = pcr[x] - 128;
*prgb++ = njClip((y + 359 * cr + 128) >> 8); // 色彩空间转换，YCbCr到RGB
*prgb++ = njClip((y - 88 * cb - 183 * cr + 128) >> 8);
*prgb++ = njClip((y + 454 * cb + 128) >> 8);
}
py += nj.comp[0].stride; // 移动YCbCr数据指针，每一行都是有stride的，所以当需要的数据都得到时，后面的就不管，直接丢弃，移动到下一行
pcb += nj.comp[1].stride;
pcr += nj.comp[2].stride;
}
} else if (nj.comp[0].width != nj.comp[0].stride) { // 如果宽度和stride都一样，什么都不用做
// grayscale -> only remove stride
unsigned char *pin = &nj.comp[0].pixels[nj.comp[0].stride];
unsigned char *pout = &nj.comp[0].pixels[nj.comp[0].width];
int y;
for (y = nj.comp[0].height - 1; y; --y) {
njCopyMem(pout, pin, nj.comp[0].width);
pin += nj.comp[0].stride;
pout += nj.comp[0].width;
}
nj.comp[0].stride = nj.comp[0].width;
}
}
void njInit(void) {
njFillMem(&nj, 0, sizeof(nj_context_t)); // 初始化nj_context_t
}
void njDone(void) {
int i;
for (i = 0; i < 3; ++i)
if (nj.comp[i].pixels) njFreeMem((void*) nj.comp[i].pixels);
if (nj.rgb) njFreeMem((void*) nj.rgb);
njInit();
}
nj_result_t njDecode(const void* jpeg, const int size) {
njDone();
nj.pos = (const unsigned char*) jpeg;
nj.size = size & 0x7FFFFFFF; // ？
if (nj.size < 2) return NJ_NO_JPEG;
if ((nj.pos[0] ^ 0xFF) | (nj.pos[1] ^ 0xD8)) return NJ_NO_JPEG; // 不以0xFFD8打头(为什么要用异或来判断？)
njSkip(2);
while (!nj.error) { // 有“错误”的时候离开
if ((nj.size < 2) || (nj.pos[0] != 0xFF)) return NJ_SYNTAX_ERROR; // 太小，或者不以0xFF打头
njSkip(2); // 移动到标签的后面(长度字段的前面)
switch (nj.pos[-1]) {
case 0xC0: njDecodeSOF(); break;
case 0xC4: njDecodeDHT(); break;
case 0xDB: njDecodeDQT(); break;
case 0xDD: njDecodeDRI(); break;
case 0xDA: njDecodeScan(); break;
case 0xFE: njSkipMarker(); break;
default:
if ((nj.pos[-1] & 0xF0) == 0xE0) // JPG0和APP0字段，目前都忽略
njSkipMarker();
else
return NJ_UNSUPPORTED;
}
}
if (nj.error != __NJ_FINISHED) return nj.error;
nj.error = NJ_OK;
njConvert();
return nj.error;
}
int njGetWidth(void) { return nj.width; }
int njGetHeight(void) { return nj.height; }
int njIsColor(void) { return (nj.ncomp != 1); }
unsigned char* njGetImage(void) { return (nj.ncomp == 1) ? nj.comp[0].pixels : nj.rgb; } // 一/三个分量
int njGetImageSize(void) { return nj.width * nj.height * nj.ncomp; }
#endif // _NJ_INCLUDE_HEADER_ONLY
炒现饭的帖子，内容均来自于网络或者书籍整理。
I)树
它是由N(N>=1)个有限结点组成一个具有层次关系的集合，它具有以下的特点：
a)每个结点有零个或多个子结点，没有子节点的节点称为叶节点(通常称为叶子)
b)没有父结点的结点称为根结点(通常称为根)
c)每一个非根结点有且只有一个父结点(没有多个引用指向同一个节点)
d)除了根结点外，每个子结点可以分为M个不相交的子树
|
r 55
o / \
o / \
t 88 3
. /|\
. / | \
. 67 7 23
l / \ \
e / \ \
a 53 5 9
f
|
Figure. A
如上图(Figure. A)，它像一棵自然界中倒生长的树，每个枝丫处就是一个节点，方向是由根到叶子，它实际上是一个有向图。
这里我们介绍下节点，节点是包含一个数据项和一个(多个)指向其他节点的引用项的数据结构/记录，通常我们用类似如下代码来表示一个节点：
1)
record SinglyLinkedNode {
next, // A reference to the next node
data // Data or reference to data
}
2)
record DoublyLinkedNode {
previous, // A reference to the previous node
next, // A reference to the next node
data // Data or reference to data
}
3)
record BinaryNode {
parent, // A reference to the parent node
left_child, // A reference to the left child node
right_child, // A reference to the right child node
data // Data or reference to data
}
结合上图(Figure. A)我们就很好理解了。
一些有关树的基本概念：
节点的度(node's degree): 子节点的个数叫做节点的度
树的度: 一棵树中，最大的节点的度称为树的度
节点层次(level): 根为第一层，根的子节点为第二层，以此类推
数的高度或深度(depth): 树中最大的节点层次
兄弟节点(siblings): 具有相同父节点的节点
森林: M(M>=0)棵互不相交的树的集合(删去一棵树的根，就得到一个森林；加上一个结点作树根，森林就变为一棵树)
有序树(ordered tree)/无序树(unodered tree): 若将树中每个结点的各子树看成是从左到右有次序的(即不能互换)，则称该树为有序树(ordered tree)；否则称为无序树(unodered tree)
还有诸如子孙/祖先/堂兄弟节点这些概念就不列举了
上图(Figure. A)当中，树的度是3，67节点的度是2，数的深度是4，55节点是根节点等等这些基本属性我们就能很快得出。
II)二叉树
每个结点最多有两个子树的有序树，通常有左子树(left subtree)和右子树(right subtree)，最大的节点的度为2，也就是二叉树的度为2。
满二叉树(full binary tree): 有时被称为proper binary tree或者2-tree或者strictly binary tree，就是每个节点有0或2个子节点。
完全二叉树(complete binary tree): 除了最后一层之外的其他层次(当然也可能包含最后一层)，每个节点都包含2个子节点，并且所有节点都尽可能靠左排列
平衡二叉树: 如果某二叉树中任意结点的左右子树的深度相差不超过1，那么它就是一棵平衡二叉树
二叉树的遍历，前序(pre-order)，中序(in-order)和后序(post-order)，前序是根左右，中序是左根右，后序是左右根。
一些有关二叉树的常用算法参看http://blog.csdn.net/luckyxiaoqiang/article/details/7518888
几个简单的例子:
这里的代码实现都是简单易懂的实现，帮助理解概念，没有做过优化！
参考资料:
http://en.wikipedia.org/wiki/Tree_(data_structure)
http://en.wikipedia.org/wiki/Node_(computer_science)
http://en.wikipedia.org/wiki/Branching_factor
http://en.wikipedia.org/wiki/Binary_tree
http://zhedahht.blog.163.com/blog/static/25411174201142733927831/