顯示具有 opensource 標籤的文章。 顯示所有文章
顯示具有 opensource 標籤的文章。 顯示所有文章

星期一, 5月 04, 2009

[Note] 送patch所學到的事

這個Note是一個寫patch的經驗,前陣子寫了一個讓curl即時壓縮加密FTP/HTTP/SFTP上傳時可以續傳的一個patch,patch本身倒是沒什麼特別的,反而是在送patch跟原作者的討論過程中學到了一些東西。patch歷時約一個月才commit進cvs,不過我覺得以open source的專案來說,這樣算挺快的。

似乎還是得把前因後果交代一下,
其實一開始是我有一個想法,
就是希望我上傳到遠端伺服器的檔案都能加密起來(順便壓縮更好),
但是我覺得在local這邊先壓縮過然後再上傳的話,就會佔Local的硬碟兩份空間,所以最好的方法是realtime壓縮加密之後上傳。這樣就不會佔用local的硬碟空間,而以目前的CPU也應該都可以做到即時傳輸。
此外我希望就是能夠彈性的選擇壓縮及加密方式,不論是bz2,gzip,pkzip或是pgp及AES都要能夠自行選擇及combine。
而且最好是不需另外的伺服器程式,以目前hosting都會提供的FTP帳號就能做的方法是最好。(我需要的不是像SFTP或FTP/TLS這樣傳輸時加密或壓縮而已,我想要的是在伺服器上的最終結果也是加密及壓縮過的)

跟lloyd討論之後,他是認為lftp+namepipe的方式可行。
另外我也找到了用pipe透過curl上傳的方法。
而這兩個方法也都驗證過確實可行,不過這兩個方法都同樣有個問題,就是沒辦法續傳,而這我認為是個應該要解決的問題。

比較了兩個解決方案後,我選擇了擁有我比較喜歡的BSD license,用法也比較彈性的curl下手修改,
curl的續傳問題大概長這樣:

gzip /mnt/2311/debian-500-i386-CD-1.iso -c | curl -T - -C -
ftp://myname:mypass@192.168.23.11/debian-500-i386-CD-1.iso.gz
** Resuming transfer from byte position 46792704
% Total % Received % Xferd Average Speed Time Time Time
% Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0
curl: (31) Could not seek stream

最主要是stdin不能直接SEEK_SET,這其實不是什麼大問題,不能SEEK_SET就SEQUENCIAL READ並bypass即可,看了一下原始程式便發現其實這段code早已大部完成了,只要略為更動即可,
於是略為修改了一下curl FTP/SFTP/HTTP上傳部份的code,驗證之後便將patch送出。
原以為送出patch之後就結束了,因為作者要收不收這也不是我能控制的,不過後來curl的作者回信說,功能是沒有問題,但原本的patch會影響到ABI與目前文件的相容性,問看看我是否能提供不修改ABI的作法,我原本還無法理會作者的意思,後來繼續討論才慢慢了解,作者的意思是看能否提供完整的ABI及定義回傳值供其他使用curl library部份的程式參考,因為curl不僅僅是一個客戶端程式curl,同時也包含一套廣泛的運用在其他程式甚至內建在程式語言中的library,libcurl。即使是改動小小的部份,也有可能不經意的影響到其他重要的程式。
了解了作者的目的之後,於是我又寫一個新的patch配合這個架構,測試一下續傳上傳沒問題之後就又再次的送出,這次作者將我的多個patch檔結合後又修改了一下包括說明文件的patch就直接commit到cvs上,不過問題來了,做make test的測試到HTTP PUT該項時沒過。
仔細的研究一下,還真的是第二次更動的code造成了問題。第三次的修改並驗證之後,這次就很快的patch就被接受了,目前已經commit在curl的cvs上。

這次寫patch得到一些經驗:

1. 寫patch時應配合原程式的架構及style,會比較容易被接受。
2. 寫完patch之後應照原程式的測試方式做一遍測試,而非只是測試自己的情況。
3. 大部份時候都要仔細考慮並聆聽來自原作者的意見,畢竟他是最清楚全部情況的人。
4. 不要懶的送patch,雖然可能會多費一些時間跟功夫,但送了patch通常可以學到更多。

ref: 我和curl作者討論的過程,https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2709004&group_id=976

星期一, 7月 14, 2008

[tips] rewrite debian/ubuntu 's lighttpd conf script from perl to python

Today I want to port lighttpd on another platform which basically a debian sarge system but without perl and dpkg package system on it. Since it's a debian based platform so I start from porting debian's binary lighttpd package, however I've found there're some perl script lays in /usr/share/lighttpd which are used when lighttpd startup.

While I can easily dump the result of perl script into a textfile,
and then startup my lighttpd correctly, I thought "maybe port it to python is not a bad idea." (since my target platform has python!), so here is the effort:
create-mime.assign.py

#!/usr/bin/python
#
# This script directly translate from debian's lighttpd perl script:
# create-mime.assign.pl
#
# Author: timchen119.at.nospam.gmail.com
# License: Public Domain
#
import sys

try:
f = open("/etc/mime.types",'r')
extensions = {}
print "mimetype.assign = ("
for line in f:
line = line.strip()
if line.startswith('#'): continue
if line != "":
splitlist = line.split()
if len(splitlist) < 2: continue
mime = splitlist[0]
for ext in splitlist[1:]:
if ext in extensions.keys(): continue
extensions[ext] = 1
print '".%s" => "%s",' % (ext,mime)
f.close()
print ")"
except Exception,e:
print e
sys.exit(1)


include-conf-enabled.py
#!/usr/bin/python
#
# This script directly translate from debian's lighttpd perl script:
# include-conf-enabled.pl
#
# Author: timchen119.at.nospam.gmail.com
# License: Public Domain
#

import os,glob

confdir = "/etc/lighttpd/"
enabled = "conf-enabled/*.conf"

os.chdir(confdir)

for file in sorted(glob.glob(enabled)):
print 'include "%s"' % file

use-ipv6.py
#!/usr/bin/python
#
# This script directly translate from ubuntu's lighttpd perl script:
# use-ipv6.pl
#
# Author: timchen119.at.nospam.gmail.com
# License: Public Domain
#

import socket

##this sometimes not accurate. (like in vserver mode)
#if socket.has_ipv6:
#

try:
if socket.socket(socket.AF_INET6,socket.SOCK_STREAM,0):
print 'server.use-ipv6 = "enable"'
except:
pass

All of these files can be found in http://kalug.linux.org.tw/~tim/lighttpd-debian-python-script/
Well something quite interesting happened when I port the debian's create-mime.assign.pl into python, It's that my python script's final result is not equivalent to perl one and has more mime types than its :
--- perlmime.txt    2008-07-14 15:29:23.000000000 +0800
+++ pymime.txt 2008-07-14 15:29:33.000000000 +0800
@@ -114,6 +114,11 @@
".dvi" => "application/x-dvi",
".rhtml" => "application/x-httpd-eruby",
".flac" => "application/x-flac",
+".pfa" => "application/x-font",
+".pfb" => "application/x-font",
+".gsf" => "application/x-font",
+".pcf" => "application/x-font",
+".pcf.Z" => "application/x-font",
".mm" => "application/x-freemind",
".gnumeric" => "application/x-gnumeric",
".sgf" => "application/x-go-sgf",
@@ -193,6 +198,11 @@
".pk" => "application/x-tex-pk",
".texinfo" => "application/x-texinfo",
".texi" => "application/x-texinfo",
+".~" => "application/x-trash",
+".%" => "application/x-trash",
+".bak" => "application/x-trash",
+".old" => "application/x-trash",
+".sik" => "application/x-trash",
".t" => "application/x-troff",
".tr" => "application/x-troff",
".roff" => "application/x-troff",
@@ -282,6 +292,7 @@
".tgf" => "chemical/x-mdl-tgf",
".mcif" => "chemical/x-mmcif",
".mol2" => "chemical/x-mol2",
+".b" => "chemical/x-molconn-Z",
".gpt" => "chemical/x-mopac-graph",
".mop" => "chemical/x-mopac-input",
".mopcrt" => "chemical/x-mopac-input",

So I start to dig why this happened, and I've found a strange perl regex filter all these mimetypes out, I believe it's a minor bug in original perl program. (or it does implicitly doing something meaningful? well I can't figure it out.)
--- create-mime.assign.pl    2008-07-14 15:35:58.000000000 +0800
+++ create-mime.assign.pl.new 2008-07-14 15:36:07.000000000 +0800
@@ -7,7 +7,7 @@
chomp;
s/\#.*//;
next if /^\w*$/;
- if(/^([a-z0-9\/+-.]+)\s+((?:[a-z0-9.+-]+[ ]?)+)$/) {
+ if(/^([A-Za-z0-9\/+-.~%]+)\s+((?:[A-Za-z0-9.+-~%]+[ ]?)+)$/) {
foreach(split / /, $2) {
# mime.types can have same extension for different
# mime types

replace this line and this will produce same results as mine.

usage:
just copy these py scripts to /usr/share/lighttpd
and change these lines if you're using debian based system
#### external configuration files
## mimetype mapping
#include_shell "/usr/share/lighttpd/create-mime.assign.pl"
include_shell "/usr/share/lighttpd/create-mime.assign.py"

## load enabled configuration files,
## read /etc/lighttpd/conf-available/README first
#include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
include_shell "/usr/share/lighttpd/include-conf-enabled.py"

星期日, 10月 21, 2007

[funny] Richard Stallman (RMS) 被忍者攻擊.

來自 Slashdot 的報導:

耶魯大學政治聯盟(Yale
Political Union, YPU,美國最古老的學生政治組織)的學生

仿效著名geek漫畫xkcd, 在耶魯一場Stallman對於DRM(Digital restrictions management)議題的演講裡, 對Stallman進行(玩笑似的)攻擊.




Richard Stallman主要在耶魯大學講述的議題為DRM,
RMS 認為現在大型商業公司(Apple,Microsoft,Sony)推動的DRM(數位著作權管理)是非法的,

常用的DRM限制包括:

* 必須在指定電腦或播放機才能播放
* 必須在特定的日期前才能播放
* 播放的次數
* 傳輸到播放機的次數
* 燒錄到光碟的次數
* 以上限制的混合
(引用自Wikipedia的DRM條目)

而GPLv3也可說是RichardStallman對DRM全面宣戰的GPL改進條款.

耶魯大學政治聯盟在RMS演講之後, 進行對 "DRM是非法的"這個議題的投票, 以33對20票通過.

不論我喜不喜歡GPL, RMS無疑是自由軟體界真正的鬥士,
他對於他眼中的惡向來從不讓步,
沒有GPL與FSF, 今日的自由軟體絕不能如此蓬勃.
而這樣具有風骨的人物跟信念堅持, 不論我是否同意他的論點, 都是讓我值得尊敬的.

星期二, 7月 03, 2007

台灣opensource社群活動行事曆

YurenJu目前正在維護台灣OpenSource社群活動行事曆, 台灣各地其實到處都有各種開放原始碼的社群聚會, 只是有時這些活動的時間不大容易記住, 也很容易排定的時間上有所衝突, 有個可以一目瞭然的行事曆的確是好的多.

目前我已在維護的kalug及python星球上加入連結,
其他社群的朋友如果想加入到自己的網站上的話,
只要插入下面這一段HTML碼就可以了.

<div>
<iframe style=" border-width:0 " width="240"
frameborder="0"
src="http://www.google.com/calendar/embed?src=p09uh8cg4uvt2ij4obf45cltsk%40group.calendar.google.com&chrome=NONE&mode=AGENDA&hl=zh_TW&height=350"
height="350">
</iframe>
<a target="_blank"
href="http://www.google.com/calendar/render?cid=p09uh8cg4uvt2ij4obf45cltsk%40group.calendar.google.com">
<img border="0"
src="http://www.google.com/calendar/images/ext/gc_button1_en.gif"/></a>
</div>

另外yurenju也在號召有興趣共同參與更新行事曆的朋友, 如果您的組織有經常性的活動的話, 可以跟yurenju聯絡.

星期五, 5月 04, 2007

[humor,hotlink] If On a Spring's Night a String ...


#!/usr/bin/env python
class tim:
def __init__(self,tim):self.chen=tim
def __rlshift__(self,chen):return tim(lambda x,z=self,tim=chen:z.chen(tim,x))
def __rshift__(self,tim):return self.chen(tim)
def __call__(self,x,tim):return self.chen(x,tim)
print "0"+hex(((reduce(lambda chen,tim: tim*chen,[int (x,32) for x in "timchen119"])
*4+119<<tim(lambda tim,chen: tim*chen+119)>>((int("119"+str(93828751754616119383408*127*119))
-1190929898972105379252210654550)))-5210475734-119))[2:-1]

I just wrote my worst python script ever.
I believe it's legal. What do you think?

星期二, 5月 01, 2007

[link] Microsoft發表Dynamic Language Runtime (DLR).

我說, 這名字對了!!! ;D

有了Jim Hugunin的加持之後,
Microsoft這次沒有笨到取一個Sxxxxt Language Runtime這種蠢名字,
看來Microsoft這次真的知道他們在作什麼. ;D

來自Jython/Ironpython作者Jim Hugunin的消息, 看來Microsoft吸收了以往對於網路不夠重視的教訓, 這次對於業界的Dynamic Language潮投注了不少心力.

消息節錄重點如下:

"""
We're initially building four languages on top of the DLR - Python, JavaScript (EcmaScript 3.0), Visual Basic and Ruby. We shipped today both Python and JavaScript as part of the Silverlight 1.1alpha1 release today.
"""

所以說DLR 目前已經支持 Python, Javascript,Visual Basic跟Ruby...

!? Python跟Ruby...!?
喔妳沒看錯... Microsoft居然開始注意Python跟Ruby的支援...
這一天終於要來了嗎?...

並且今天釋出的Silverlight 1.1alpha1版已支援python跟Javascript. (Silverlight看起來將會是下一代Microsoft主推的web平台??)

"""
In addition to the Silverlight release, we've also made the full source code for both IronPython and all of the new DLR platform code available on codeplex under the BSD-style Microsoft Permissive License.
"""

BSD-style Microsoft Permissive License !???

BSD style?!沒聽錯吧... 會不會..太opensource了一點? :DD

看來我這文章標題應該改成==> 神奇的Jim Hugunin
,Microsoft的自由軟體跟Python/Dynamic Language推手... XD

雖然我常常沒事虧一下MS,
不過這次還是得替Jim Hugunin跟Microsoft鼓鼓掌...
在MS裡搞自由軟體... 讚啊...