brillo-m9-dev/brillo-m9-release版本release出来已经有一段时间了,之前一直在学习OpenWrt 15.05就暂停了一段时间。webservd是Brillo系统与外界交互的模块之一,我们之前一直使用curl命令去获取设备信息,进行认证以获取access token,进行wifi连接配置以及发送命令,都是通过webservd来传达的。webservd会通过external/libminihttp来建立一个简单的http/https server。
system/webservd(@ffc49cf)下面主要包含这两个模块:
- webservd
- libwebserv
这两个模块是完全独立的:libwebserv其实是weaved(system/weaved)的一下模块,是webservd的一个客户端。由于webservd需要通过dbus访问其他的服务如firewalld, keystore等。libwebserv与webservd也是通过dbus进行通信的。所以代码中还包含dbus binding相关的两个模块:
- libwebservd-client-internal: 由system/webservd/webservd/dbus_bindings/文件夹下的文件生成
- libwebserv-proxies-internal: 由system/webservd/libwebservd/dbus_bindings/文件夹下的文件生成
- webservd – 文件结构
webservd/ ├── Android.mk ├── config.cc ├── config.h ├── config_unittest.cc ├── dbus_bindings │ ├── dbus-service-config.json │ ├── org.chromium.WebServer.ProtocolHandler.dbus-xml │ └── org.chromium.WebServer.Server.dbus-xml ├── dbus_protocol_handler.cc ├── dbus_protocol_handler.h ├── dbus_request_handler.cc ├── dbus_request_handler.h ├── encryptor.h ├── error_codes.cc ├── error_codes.h ├── etc │ ├── dbus-1 │ │ └── org.chromium.WebServer.conf │ └── init │ └── webservd.conf ├── fake_encryptor.cc ├── fake_encryptor.h ├── firewalld_firewall.cc ├── firewalld_firewall.h ├── firewall_interface.h ├── keystore_encryptor.cc ├── keystore_encryptor.h ├── log_manager.cc ├── log_manager.h ├── log_manager_unittest.cc ├── main.cc ├── permission_broker_firewall.cc ├── permission_broker_firewall.h ├── protocol_handler.cc ├── protocol_handler.h ├── request.cc ├── request.h ├── request_handler_interface.h ├── server.cc ├── server.h ├── server_interface.h ├── temp_file_manager.cc ├── temp_file_manager.h ├── utils.cc ├── utils.h ├── webservd.rc └── webservd_testrunner.cc
从文件名称可以看出webservd包含配置相关(config)、dbus进程间通信(dbus_xxx)、防火墙设置(firewall)、加密认证(keystore)、log管理、http/https协议请求处理、建立http/https服务、通过http/https协议上传文件管理(temp_file_manager)以及init配置文件。
其他文件:
- unit test相关的文件: config_unittest.cc, webservd_testrunner.cc
- 早期使用md5对RSA private key进行加密解密的代码: fake_encryptor.cc
- chromeos中设置firewall相关的代码: permission_broker_firewall.cc
- webservd – 配置相关(config.cc)
主要包含:
- protocol handler name: “http”, “https”
- port number: default: 80(http), 443(https)
- tls(Transport Layer Security): https(true), http(false)
- interface name: 如eth0, wlan0
- RSA private key: 配置https service
- certificate: 配置https service
- certificate fingerprint
- debug: 通过命令行参数进行配置
- use ipv6: 通过命令行参数进行配置
- log directory
- default request timeout: 60 seconds
web server的默认配置如下:
// Default configuration for the web server. const char kDefaultConfig[] = R"({ "protocol_handlers": [ { "name": "http", "port": 80, "use_tls": false }, { "name": "https", "port": 443, "use_tls": true } ] })";
在web server默认配置中,定义了两个protocol handler: http & https。web server配置也可通过命令行参数进行指定。
- webservd – 启动流程
先看一下system/webservd/webservd/webservd.rc文件:
on post-fs-data mkdir /data/misc/webservd 0700 webserv webserv mkdir /data/misc/webservd/logs 0700 webserv webserv mkdir /data/misc/webservd/uploads 0700 webserv webserv service webservd /system/bin/webservd class late_start user webserv group webserv dbus inet
init进行在post-fs-data阶段,会在/data/misc/webservd下面创建:
- logs – 存放log
- uploads – 通过web server上传的文件去先存放在这里
同时可以看出webservd以webserv的用户、webserv dbus inet的组运行。
然后就到了main.cc:
在main函数里,会先处理命令行参数:
- log_to_stderr: 是否将log输出到stderr,默认为false
- config_path: web server的配置文件,默认为空,使用default configuration
- debug: 是否在web请求中打印debug信息,默认为false
- ipv6: 是否要支持ipv6, 默认为true
再接着加载配置文件:
const char kDefaultConfigFilePath[] = "/etc/webservd/config"; // ... int main(int argc, char* argv[]) { // ... base::FilePath default_file_path{kDefaultConfigFilePath}; if (!FLAGS_config_path.empty()) { // In tests, we'll override the board specific and default configurations // with a test specific configuration. webservd::LoadConfigFromFile(base::FilePath{FLAGS_config_path}, &config); } else if (base::PathExists(default_file_path)) { // Some boards have a configuration they will want to use to override // our defaults. Part of our interface is to look for this in a // standard location. CHECK(webservd::LoadConfigFromFile(default_file_path, &config)); } else { webservd::LoadDefaultConfig(&config); } // ... }
可以看到优先加载命令行指定的配置文件,再接着从/etc/webservd/config加载配置文件,最后才使用默认的配置。
再接着就是根据配置信息创建network interface socket, network interface socket可以绑定到特定的network interface(如eth0, wlan0),如果不指定,就可以接收任意network interface的请求:
int CreateNetworkInterfaceSocket(const std::string& if_name) {
// The following is basically the steps libmicrohttpd normally takes
// when creating a new listening socket and binding it to a port.
int socket_fd = socket(PF_INET6, SOCK_STREAM | SOCK_CLOEXEC, 0);
if (socket_fd < 0 && errno == EINVAL)
socket_fd = socket(PF_INET6, SOCK_STREAM, 0);
if (socket_fd < 0) {
PLOG(ERROR) << "Unable to create a listening socket";
return -1;
}
// Now, specify that we want this socket to bind only to a particular
// network interface. Note, this call requires root privileges, so this
// should be done before the privileges are dropped.
if (setsockopt(socket_fd, SOL_SOCKET, SO_BINDTODEVICE,
if_name.c_str(), if_name.size()) < 0) {
PLOG(WARNING) << "Failed to bind socket (SO_BINDTODEVICE) to " << if_name;
close(socket_fd);
return -1;
}
return socket_fd;
}
最后执行brillo::Daemon::Run() @ system/webservd/webservd/main.cc:
-> main() @ system/webservd/webservd/main.cc -> brillo::Daemon::Run() @ external/librillo/brillo/daemons/daemon.cc -> brillo::DBusServiceDaemon::OnInit() @ external/libbrillo/brillo/daemons/dbus_daemon.cc -> (anonymous namespace)::Daemon::RegisterDBusObjectsAsync() @ system/webservd/webservd/main.cc
在(anonymous namespace)::Daemon::RegisterDBusObjectsAsync()中:
-> 初始化log manager
-> 创建firewall service client
-> 创建webservd::Server对象
` -> 创建dbus object
` -> 创建default encryptor
` -> 设置default request timeout
->执行webservd::RegisterAsync()
` -> 初始化tls
` -> 创建protocol handler
` -> 注册firewall service online callback: webservd::FirewalldFirewall::WaitForServiceAsync()
` -> export Server
` -> export ProtocolHandler
- webservd – log管理
由system/webservd/webservd/log_manager.cc可以看出最多可以有7个log文件,文件名以日期的形式命名,如2015-02-25.log,单个文件最大可以到1MB, 如果文件大于1MB, 那么会被重命名为日期+[a-z].log的形式,如2015-02-15-a.log的形式。
log记录的web客户端ip地址,服务器响应请求的时间,方法(GET, POST, PUT, PATCH),访问的url, http版本号,响应的状态码以及大小, 如:
127.0.0.1 - - [25/Feb/2015:03:29:12 -0800] "GET /test HTTP/1.1" 200 2326
- webservd – 证书 ->@<-
建立https服务是需要证书的,brilloemulator使用的是自签名证书。创建证书的过程如下:
-> 创建X509 v3证书: CN=”Brillo device”, 有效期~5年
-> 创建RSA key pair: key length: 1024bits
-> 保存RSA private key到temp buffer
-> 创建EVP: The Digital EnVeloPe library @ openssl-1.0.1r/doc/ssleay.txt
-> EVP: assign RSA private key
-> EVP: 证书 + RSA public key
-> 证书签名
-> 将签名后的证书保存到/data/misc/webservd/certificate文件中
-> 使用key store对RSA private key进行加密
-> 将加密后的RSA private key保存到/data/misc/webservd/key文件中
调用的流程如下:
-> main() @ system/webservd/webservd/main.cc -> brillo::Daemon::Run() @ external/libbrillo/brillo/daemons/daemon.cc -> brillo::DBusServiceDaemon::OnInit() @ external/libbrillo/brillo/daemons/dbus_daemon.cc -> (anonymous namespace)::Daemon::RegisterDBusObjectsAsync() @ system/webservd/webservd/main.cc -> webservd::Server::RegisterAsync() @ system/webservd/webservd/server.cc -> webservd::Server::InitTlsData() @ system/webservd/webservd/server.cc
- webservd – 建立http/https服务
-> main() @ system/webservd/webservd/main.cc -> brillo::Daemon::Run() @ external/libbrillo/brillo/daemons/daemon.cc -> brillo::DBusServiceDaemon::OnInit() @ external/libbrillo/brillo/daemons/dbus_daemon.cc -> (anonymous namespace)::Daemon::RegisterDBusObjectsAsync() @ system/webservd/webservd/main.cc -> webservd::Server::RegisterAsync() @ system/webservd/webservd/server.cc -> webservd::Server::CreateProtocolHandler() @ system/webservd/webservd/server.cc -> webservd::ProtocolHandler::Start() @ system/webservd/webservd/protocol_handler.cc -> webservd::ProtocolHandler::DoWork() @ system/webservd/webservd/protocol_handler.cc
- webservd – 防火墙配置
建立web server时,必然需要开放端口供客户端访问,Brillo系统使用的网络防火墙由iptable来实现,默认的配置如下(由system/etc/init.firewall-setup.sh完成):
#!/system/bin/sh # # Set up default firewall rules. BINPATH=/system/bin # IPv4 only rules. iptables_icmp_setup() { ${BINPATH}/iptables -A INPUT -p icmp -j ACCEPT -w } iptables_mdns_setup() { ${BINPATH}/iptables -A INPUT -p udp --destination 224.0.0.251 --dport 5353 -j ACCEPT -w } # IPv6 only rules. ip6tables_icmp_setup() { ${BINPATH}/ip6tables -A INPUT -p ipv6-icmp -j ACCEPT -w # Allow all outbound ICMPv6 traffic. This is important for things like # neighbor discovery and address negotiation. ${BINPATH}/ip6tables -A OUTPUT -p ipv6-icmp -j ACCEPT -w } ip6tables_mdns_setup() { ${BINPATH}/ip6tables -A INPUT -p udp --destination FF02::FB --dport 5353 -j ACCEPT -w } # Install all IPv4 and IPv6 rules. for iptables in ip{,6}tables; do iptables_bin=${BINPATH}/${iptables} [ -x ${iptables_bin} ] || continue # Set default policy to DROP. ${iptables_bin} -P INPUT DROP -w ${iptables_bin} -P FORWARD DROP -w ${iptables_bin} -P OUTPUT DROP -w # Accept everything on the loopback. ${iptables_bin} -I INPUT -i lo -j ACCEPT -w ${iptables_bin} -I OUTPUT -o lo -j ACCEPT -w # Accept return traffic inbound. ${iptables_bin} -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -w # Accept icmp echo (NB: icmp echo ratelimiting is done by the kernel). ${iptables}_icmp_setup # Accept new and return traffic outbound. ${iptables_bin} -I OUTPUT -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT -w # Accept inbound mDNS traffic. ${iptables}_mdns_setup # Accept DHCP traffic (communicating as either client or server). ${iptables_bin} -I INPUT -p udp --dport 67:68 --sport 67:68 -j ACCEPT -w ${iptables_bin} -I OUTPUT -p udp --dport 67:68 --sport 67:68 -j ACCEPT -w done # Set completion property. setprop firewall.init 1
使用iptables命令查看可能会更直观一些:
hzak@B85PRO:~$ adb shell iptables --list Chain INPUT (policy DROP) target prot opt source destination ACCEPT tcp -- anywhere anywhere tcp dpt:https ACCEPT tcp -- anywhere anywhere tcp dpt:http ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc dpts:bootps:bootpc ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere tcp dpt:5555 ACCEPT icmp -- anywhere anywhere ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy DROP) target prot opt source destination ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc dpts:bootps:bootpc ACCEPT all -- anywhere anywhere state NEW,RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere hzak@B85PRO:~$ adb shell ip6tables --list Chain INPUT (policy DROP) target prot opt source destination ACCEPT udp anywhere anywhere udp spts:bootps:bootpc dpts:bootps:bootpc ACCEPT all anywhere anywhere state RELATED,ESTABLISHED ACCEPT all anywhere anywhere ACCEPT icmpv6 anywhere anywhere ACCEPT udp anywhere ff02::fb udp dpt:mdns Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy DROP) target prot opt source destination ACCEPT udp anywhere anywhere udp spts:bootps:bootpc dpts:bootps:bootpc ACCEPT all anywhere anywhere state NEW,RELATED,ESTABLISHED ACCEPT all anywhere anywhere ACCEPT icmpv6 anywhere anywhere
由于注册过firewall service online callback, 所以当firewall service online的时候,会回调:
void Server::OnFirewallServiceOnline() { LOG(INFO) << "Firewall service is on-line. " << "Opening firewall for protocol handlers"; for (auto& handler_config : config_.protocol_handlers) { VLOG(1) << "Firewall request: Protocol Handler = " << handler_config.name << ", Port = " << handler_config.port << ", Interface = " << handler_config.interface_name; firewall_->PunchTcpHoleAsync( handler_config.port, handler_config.interface_name, base::Bind(&OnFirewallSuccess, handler_config.interface_name, handler_config.port), base::Bind(&IgnoreFirewallDBusMethodError)); } }
void FirewalldFirewall::PunchTcpHoleAsync( uint16_t port, const std::string& interface_name, const base::Callback<void(bool)>& success_cb, const base::Callback<void(brillo::Error*)>& failure_cb) { proxy_->PunchTcpHoleAsync(port, interface_name, success_cb, failure_cb); }
去更新防火墙的设置,如打开80端口(http)
- webservd – Add request handler
Add request handler是由weaved (system/weaved)发起的,向webservd注册web API。调用的过程如下:
-> main() @ system/webservd/webservd/main.cc -> brillo::Daemon::Run() @ external/libbrillo/brillo/daemons/daemon.cc -> brillo::BaseMessageLoop::Run() @ external/libbrillo/brillo/daemons/daemon.cc -> base::RunLoop::Run() @ external/libchrome/base/run_loop.cc -> base::MessagePumpLibevent::Run() @ external/libchrome/bae/message_loop/message_pump_libevent.cc -> base::MessageLoop::DoWork() @ external/libchrome/base/message_loop/message_loop.cc -> base::MessageLoop::DeferOrRunPendingTask() @ external/libchrome/base/message_loop/message_loop.cc -> base::MessageLoop::RunTask() @ external/libchrome/base/message/message_loop.cc -> base::debug::TaskAnnotator::RunTask() @ external/libchrome/base/debug/task_annotator.cc -> base::Callback<>::Run() @ external/libchrome/base/callback.h -> dbus::Bus::ProcessAllIncomingDataIfAny() @ external/libchrome/dbus/bus.cc -> dbus_connection_dispatch() @ external/dbus/dbus/dbus-connection.cc -> _dbus_object_tree_dispatch_and_unlock() @ external/dbus/dbus/dbus-object-tree.c -> dbus::ExportedObject::HandleMessage() @ external/libchrome/dbus/exported_object.cc -> base::Callback<>::Run() @ external/libchrome/base/callback.h -> base::internal::Invoker<>::Run() @ external/libchrome/base/bind_internal.h -> base::internal::InvokeHelper<>::MakeItSo() @ external/libchrome/base/bind_internal.h -> base::internal::RunnableAdapter<>::Run() @ external/libchrome/base/bind_internal.h -> brillo::dbus_utils::SimpleDBusInterfaceMethodHandlerWithErrorAndMessage() @ external/libbrillo/brillo/dbus/dbus_oject.cc -> brillo::dbus_utils::DBusParamReader<>::Invoke() @ external/libbrillo/brillo/dbus/dbus_param_reader.h -> brillo::dbus_utils::DBusParamReader<>::InvokeHelper() @ external/libbrillo/brillo/dbus/dbus_param_reader.h -> brillo::dbus_utils::DBusParamReader<>::Invoke() @ external/libbrillo/brillo/dbus/dbus_param_reader.h -> brillo::dbus_utils::DBusParamReader<>::InvokeHelper() @ external/libbrillo/brillo/dbus/dbus_param_reader.h -> brillo::dbus_utils::DBusParamReader<>::Invoke() @ external/libbrillo/brillo/dbus/dbus_param_reader.h -> brillo::dbus_utils::DBusParamReader<>::InvokeHelper() @ external/libbrillo/brillo/dbus/dbus_param_reader.h -> brillo::dbus_utils::DBusParamReader<>::Invoke() @ external/libbrillo/brillo/dbus/dbus_param_reader.h -> brillo::dbus_utils::SimpleDBusInterfaceMethodHandlerWithErrorAndMessage<>::HandleMethod() -> base::Callback<>::Run() @ external/libchrome/base/callback.h -> base::internal::Invoke<>::Run() @ external/libchrome/base/bind_internal.h -> base::internal::InvokeHelper<>::MakeItSo() @ external/libchrome/base/bind_internal.h -> base::internal::RunnableAdapter<>::Run() @ external/libchrome/base/bind_internal.h -> webservd::DBusProtocolHandler::AddRequestHandler() @ system/webservd/webservd/dbus_protocol_handler.cc -> webservd::ProtocolHandler::AddRequestHandler() @ system/webservd/webservd/protocol_handler.cc
我们再看一下webservd::ProtocolHandler::AddRequestHandler()的实现:
std::string ProtocolHandler::AddRequestHandler( const std::string& url, const std::string& method, std::unique_ptr<RequestHandlerInterface> handler) { std::string handler_id = base::GenerateGUID(); request_handlers_.emplace(handler_id, HandlerMapEntry{url, method, std::move(handler)}); return handler_id; }
- webservd – 处理请求(handle request)
当我们在客户端执行命令时,webservd就会将请求通过D-Bus转发给weaved(system/weaved)。如我们执行如下命令:
$ curl -H "Authorization: Privet anonymous" -k https://localhost:8443/privet/info
webservd内部调用的流程如下:
-> main() @ system/webservd/webservd/main.cc -> brillo::Daemon::Run() @ external/libbrillo/brillo/daemons/daemon.cc -> brillo::BaseMessageLoop::Run() @ external/libbrillo/brillo/message_loops/base_message_loop.cc -> base::RunLoop() @ external/libchrome/base/run_loop.cc -> base::MessagePumpLibevent::Run() @ external/libchrome/base/message_loop/message_pump_libevent.cc -> base::MessageLoop::DoWork() @ external/libchrome/base/message_loop/message_loop.cc -> base::MessageLoop::DeferOrRunPendingTask() @ external/libchrome/base/message_loop/message_loop.cc -> base::MessageLoop:RunTask() @ external/libchrome/base/message_loop/message_loop.cc -> base::debug::TaskAnnotator::RunTask() @ external/libchrome/base/debug/task_annotator.cc -> base::Callback<void ()>::Run() @ external/libchrome/base/callback.h -> webservd::ProtocolHandler::DoWork() @ system/webservd/webservd/protocol_handler.cc -> MHD_run() @ external/libmicrohttpd/src/microhttpd/daemon.c -> MHD_select() @ external/libmicrohttpd/src/microhttpd/daemon.c -> MHD_run_from_select() @ external/libmicrohttpd/src/microhttpd/daemon.c -> MHD_connection_handle_idle() @ external/libmicrohttpd/src/microhttpd/connection.c -> call_connection_handler() @ external/libmicrohttpd/src/microhttpd/connection.c -> webservd::ServerHelper::ConnectionHandler() @ system/webservd/webservd/protocol_handler.cc -> webservd::Request::BeginRequestData() @ system/webservd/webservd/request.cc -> webservd::ForwardRequestToHandler() @ system/webservd/webservd/request.cc // Send the request over D-Bus and await the response
从这里可以看到webservd中用到了libmicrohttpd,Message loop和D-Bus。
- webservd – 上传文件
如果上传的数据中需要使用到文件,可以以表单的形式上传文件到Brillo系统主机。上传文件的命令如下:
$ curl -H "Authorization: Privet anonymous" -X POST -F update=@configure -k https://localhost:8443/privet/info
这里,update是key, 而configure是value, 对应的是上传的文件名。
服务器缓存的文件名的格式如下(<guid>-<counter>):
B515AE30-1FA3-4ABA-80A5-8EB2A409AB73-1
当请求结束后,上传的文件会被删除。