This blog is rated 🔞, viewer discretion is advised

在py3里同步/异步混合使用 httpx 调用

比如开发需求是请求一个 http API,得到数据,解析一下返回,那么一般的做法是封装一个方法,比如

import httpx


def get_sth(p1, default=MY_VAL):
    # network
    r = httpx.get(API_URL, params={'t1': p1})
    # parsing
    res = r.json().get('my_key') or MY_VAL

但是如果想在 async/await 里用这段代码,就得改成

import httpx


async def get_sth(p1, default=MY_VAL):
    # network
    with http.AsyncClient() as client:
        r = await client.get(API_URL, params={'t1': p1})
    # parsing
    res = r.json().get('my_key') or MY_VAL

注意其中 def get_sth() 也必须改成 async def get_sth 。这就是所谓 async/await 传染性

这个时候,如果你想把这块代码抽象出来,让同步/异步的库都能调用,我在最近重构里找到一个最佳实践:

import httpx


def get_sth(p1, default=MY_VAL):
    r = httpx.Request('GET', API_URL, params={'t1': p1})
    r.parse = lambda x: (x.get('my_key') or MY_VAL)

# 同步调用:
with httpx.Client() as client:
    r = client.send(get_sth())
r.request.parse(r.json())

# 异步调用:
async with httpx.AsyncClient() as client:
    r = await client.send(get_sth())
r.request.parse(r.json())

思路是 逻辑和 transport解耦,也是某种意义上的 Sans-IO 了吧?

Posted

stdout

git切换马甲

如果你有多个 github、bitbucket、gitlab 账号需要来回切换,那么这个方法或许对你有用。

使用git你需要配置一个 ~/.ssh/id_rsa,如果你有多个身份就麻烦了。传统网上搜到的办法是编辑 ~/.ssh/config 然后加入个类似这样的配置:

  Host estgit
  HostName github.com
  Port 22
  User git
  IdentityFile ~/.ssh/est_github
  IdentitiesOnly yes

这样做也不是不行,就是你输入命令的时候得把主机名改了。比如

git clone git@github.com:est/aether-pelican.git

你需要改成

git@estgit:est/aether-pelican.git

当然这还不是最蛋痛的。最蛋痛的是如果你用了 git submodules 那么你切换马甲变得非常麻烦。因为别人用你的项目没法解析 git@estgit 这样的玩意。

我研究出来了一个hack,直接改repo的 .git/config 可以达到同样的效果。比如

  [core]
      sshCommand = /usr/bin/ssh -o IdentitiesOnly=yes -i ~/.ssh/est_github -a
  [user]
      name = est
      email = ...@...

这样你可以随意在不同的repo切换n个 ssh identity 身份。

Posted

stdout

Rimworld 1.4 Biotech Xenogene/Endogene code names

I found out the corresponding gene code by accident, for Yuran Race mod it's located under

~/Library/Application\ Support/Steam/SteamApps/workshop/content/*/2844129100/Bio1.4/Defs/YuranBioTitle.xml

So the official description can be found at (or wiki):

~/Library/Application\ Support/Steam/SteamApps/common/RimWorld/RimWorldMac.app/Data/Biotech/Defs/GeneDefs/

I need to use the dev tool to modify gene of my pawn, so here's the table:

import os
import xml.etree.ElementTree as ET
p = os.path.expanduser('~/Library/Application Support/Steam/SteamApps/common/RimWorld/RimWorldMac.app/Data/Biotech/Defs/GeneDefs/')
fields = 'defName label description biostatCpx biostatMet biostatArc'.split()
print('| %s |' % ' | '.join(fields))
print('|--------' * len(fields) + '|')
for f in os.listdir(p):
    root = ET.fromstring(open(p + f).read())
    for elem in root.findall('GeneDef'):
        vals = [t.text if isinstance(t:=elem.find(x), ET.Element) else '' for x in fields]
        if vals[0]:
            print('| %s |' % ' | '.join(vals))

The result is:

defName label description biostatCpx biostatMet biostatArc
Hair_BaldOnly no hair Carriers of this gene grow no hair on the head.
Hair_ShortOnly short-haired Carriers of this gene can only grow short hair.
Hair_LongOnly long-haired Carriers of this gene grow hair on the head very quickly.
Hair_Grayless grayless hair Carriers of this gene keep their natural hair color as they age.
Beard_BushyOnly only bushy beards Male carriers of this gene experience rapid beard growth and are uncomfortable cutting their beards.
Beard_NoBeardOnly beardless Carriers of this gene grow no facial hair.
Beard_Always unisex beards Carriers of this gene always have thick facial hair, even women.
Skin_InkBlack ink black skin Carriers of this gene produce a pigment that turns their skin a pale black color almost as dark as ink.
Skin_SlateGray slate gray skin Carriers of this gene produce a pigment that turns their skin slate gray.
Skin_LightGray light gray skin Carriers of this produce a light-gray pigment in their skin.
Skin_SheerWhite sheer white skin Carriers of this gene have sheer white skin, unlike natural skin tones, due to a special engineered reflective cell covering.
Skin_Blue blue skin Carriers of this gene produce a pigment that turns their skin a blue color.
Skin_Purple purple skin Carriers of this gene produce a pigment that gives their skin a purple color.
Skin_PaleRed pale red skin Carriers of this gene produce a pigment that turns their skin a moderate red color.
Skin_DeepRed deep red skin Carriers of this gene produce a deep-red pigment that gives their skin an almost bloody appearance.
Skin_PaleYellow pale yellow skin Carriers of this gene produce a pigment that turns their skin a grayish yellow color.
Skin_DeepYellow deep yellow skin Carriers of this gene produce a pigment that gives their skin a deep yellow color.
Skin_Orange orange skin Carriers of this gene produce a pigment that gives their skin an orange color.
Skin_Green green skin Carriers of this gene produce a pigment that gives their skin a green color.
Furskin furskin Carriers of this gene grow thick fur all over their body, which protects them from cold temperatures. 1 -1
Eyes_Red red eyes Carriers of this gene have deeply red-pigmented eyes.
Eyes_Gray gray eyes Carriers of this gene have pale white-gray eyes.
Brow_Heavy heavy brow Carriers of this gene have a prominent brow. 0
Tail_Furry furry tail Carriers of this gene grow a fluffy tail which partially protects them from cold temperatures. 1 -1
Tail_Smooth smooth tail Carriers of this gene grow a slender tail that can act as a dexterous fifth limb. 1 -1
FacialRidges facial ridges Carriers of this gene grow raised ridges of skin on their face. 0
MinTemp_SmallIncrease cold weakness Carriers of this gene are slightly less comfortable in cold temperatures. 1
MinTemp_SmallDecrease cold tolerant Carriers of this gene are slightly more comfortable in cold temperatures. -1
MinTemp_LargeDecrease cold super-tolerant Carriers of this gene are much more comfortable in cold temperatures. -2
MaxTemp_LargeIncrease heat super-tolerant Carriers of this gene are more comfortable in warm temperatures. -2
MaxTemp_SmallIncrease heat tolerant Carriers of this gene are slightly more comfortable in warm temperatures. -1
MaxTemp_SmallDecrease heat weakness Carriers of this gene are slightly less comfortable in warm temperatures. 1
PsychicAbility_Deaf psychically deaf Carriers of this gene are deaf to all psychic energy and influence outside their own minds. They cannot be affected by psychic influence, nor can they ever wield psychic power. 2
PsychicAbility_Dull psychically dull Carriers of this gene are less psychically-sensitive than others. 1
PsychicAbility_Enhanced psy-sensitive Carriers of this gene are more psychically-sensitive than average. -2
PsychicAbility_Extreme super psy-sensitive Carriers of this gene are much more psychically-sensitive than most. 2 -5
MoveSpeed_Slow slow runner Carriers of this gene move more slowly than normal. 3
MoveSpeed_Quick fast runner Carriers of this gene move more quickly than normal. -3
MoveSpeed_VeryQuick very fast runner Carriers of this gene move much more quickly than normal. -5
Beauty_VeryUgly very unattractive Carriers of this gene have misshapen, asymmetrical facial structures and blotchy skin. They're hard to look at. 2
Beauty_Ugly unattractive Carriers of this gene have exaggerated facial features and poor skin that are generally considered ugly. 1
Beauty_Pretty attractive Carriers of this gene have unusually symmetrical, balanced facial features and extra-clear skin which gives them a pleasing appearance. -1
Beauty_Beautiful very attractive Carriers of this gene have remarkably precise and symmetrical faces. Their features are distinctive and strong without being exaggerated, and their skin is nearly perfect. They are generally seen as beautiful. -2
Learning_Slow slow study Carriers of this gene have deficient long-term memories and don't understand new ideas quickly. They are slow at learning new skills and knowledge. 2
Learning_Fast quick study Carriers of this gene have excellent memories and grasp new ideas quickly. They learn faster than others. 2 -3
Mood_Depressive very unhappy Carriers of this gene are highly predisposed to negative emotion. They'll see the bad in every situation and have a much lower mood than others. 5
Mood_Pessimist unhappy Carriers of this gene are predisposed to pessimistic perceptions. They'll tend to interpret things negatively and have lower mood than others. 3
Mood_Optimist happy Carriers of this gene are predisposed to optimistic feelings. They'll have higher mood than others. 2 -1
Mood_Sanguine very happy Carriers of this gene are highly predisposed to optimism and not at all inclined to think negatively. They'll have much higher mood than others. 3 -2
ToxResist_Partial tox resistance Carriers of this gene are resistant to toxic buildup from any source. This includes pollution, toxic fallout, tox gas, and direct attacks with venom or injected poison. They'll gain half the amount of toxic buildup compared to others.\n\nCellular filters in the lung and skin reduce the dose of toxins entering the bloodstream. -2
ToxResist_Total tox immunity Carriers of this gene are totally immune to toxic buildup from all sources including polluted terrain, toxic fallout, tox gas, and direct attacks with venom or injected poison. They are also not bothered by acidic smog.\n\nThe carrier's biochemical pathways are modified to route around interference from nearly all known toxins. Along with enhancements to the kidneys and liver, this keeps carriers comfortable in even the most toxic of environments. 2 -4
Delicate delicate Carriers of this gene take greater injuries than others from the same damage. They have thin, brittle bones and less binding molecules in joints and flesh. 3
Robust robust Carriers of this gene take less injuries than others from the same damage. They have thickened, densified bones, nearly-solid ribcages, and strengthened binding factors in joints and flesh. -2
Pain_Reduced reduced pain Carriers of this gene feel half as much pain compared to a baseliner. Reduced neuron activity in the brain's nociception centers makes pain dull and faint. This can be advantageous sometimes, and dangerous other times. -1
Pain_Extra extra pain Carriers of this gene feel more pain than others given the same injuries. Neuron activity in the brain's nociception center is amplified, so pain feels extra-intense and fiery. This can be protective, but overall it's considered a negative and makes it hard to push through difficult situations. 2
Aggression_DeadCalm dead calm Carriers of this gene feel calm in every situation and have a very placid demeanor. They will never start social fights or have aggressive mental breaks. -1
Aggression_Aggressive aggressive Carriers of this gene are quick to anger. They are twice as likely to start social fights. When they have mental breaks, they are twice as likely to choose an aggressive kind of break. 2
Aggression_HyperAggressive hyper-aggressive Carriers of this gene are hormonally high-strung and very aggressive. They are three times as likely to start social fights. Any mental break they have will be of an aggressive type. 3
VerySleepy very sleepy Carriers of this gene get tired much faster than others. 4
Sleepy sleepy Carriers of this gene get tired somewhat faster than others. 2
LowSleep low sleep Carriers of this gene get tired less quickly than others. 2 -4
Neversleep never sleep Carriers of this gene have a unique metabolic process which allows clusters of neurons to sleep while the rest of the brain stays awake. They never need to sleep. 3 -6
MeleeDamage_Weak weak melee damage Carriers of this gene do less damage in close-quarters combat. Weak fast-twitch muscle fibers make their strikes shaky and weak. 1
MeleeDamage_Strong strong melee damage Carriers of this gene do more damage in close-quarters combat. Extra-strong fast-twitch muscle fibers make their strikes accurate and powerful. -2
UVSensitivity_Mild mild UV sensitivity Carriers of this gene have biological compounds in their skin that react painfully to UV radiation. They are unusually sensitive to sunlight. 3
UVSensitivity_Intense intense UV sensitivity Carriers of this gene have biological compounds in their skin that react dangerously to UV radiation. They are intensely sensitive to sunlight. 2 4
Libido_Low low libido Carriers of this gene are less likely to engage in lovin' with their partner.
Libido_High high libido Carriers of this gene are more likely to engage in lovin' with their partner.
FireSpew fire spew Carriers are able to spew flammable bile generated by a special organ in their neck. The bile sticks to anything in a small area and can ignite people, objects, and the ground. -2
FoamSpray foam spray Carriers grow glands in the neck that generate and store a fire-retardant foam. They can spew this foam over an area to extinguish fires. -2
LongjumpLegs longjump legs Carriers have special hemogen-powered muscle fibers in their legs which allow them to jump great distances. -2
AnimalWarcall animal warcall Carriers of this gene can perform an animal warcall, using a powerful bellow and psychic connection to call an animal to fight for them. 1 -3
Bloodfeeder bloodfeeder Carriers of this gene have small retractable fangs and an organ on the roof of the mouth which can extract hemogen from fresh warm blood. They can bite an unresisting person, suck the blood, and gain hemogen directly. -1
Coagulate coagulate Carriers of this gene have special glands on their hands and wrists, as well as a unique salivary compound that they can use to rapidly tend wounds. -1
XenogermReimplanter gene implanter Carriers of this gene can implant a copy of their xenogerm into another person through a somewhat gross-looking injector organ. Their own genetic material will then regrow very slowly. If they implant while their genes are regrowing, they will die. Germline genes will be unaffected. 3 1
PiercingSpine piercing spine Carriers grow an opening in their upper chest along with a quiver of keratin spines. Using a hemogen-powered chemical reaction, they can fire these spines at high speed at nearby targets with surprising accuracy. -1
AcidSpray acid spray Carriers grow glands in the neck that generate and store a sticky acid substance, along with acid-tolerant tissues in the mouth. They can spew this acid over an area, where it will stick to enemies and burn them over time. -2
Body_Fat fat body Carriers can have fat bodies. A person can have more than one body type gene; one body type will be chosen among those that are allowed.
Body_Thin thin body Carriers can have thin bodies. A person can have more than one body type gene; one body type will be chosen among those that are allowed.
Body_Hulk hulk body Carriers can have large bodies. A person can have more than one body type gene; one body type will be chosen among those that are allowed.
Body_Standard standard body Carriers can have average-shaped bodies. A person can have more than one body type gene; one body type will be chosen among those that are allowed.
Ears_Human human ears Carriers of this gene have regular human ears.
Ears_Pig pig ears Carriers of this gene will grow pointed pig-like ears.
Ears_Floppy floppy ears Carriers of this gene grow long, floppy hound-like ears.
Ears_Cat cat ears Carriers of this gene have cat-like ears.
Ears_Pointed pointed ears Carriers of this gene have pointed ears.
Nose_Human human nose Carriers of this gene have regular human noses.
Nose_Pig pig nose Carriers of this gene have pig-like snouts.
Jaw_Baseline human jaw Carriers of this gene have regularly-shaped jaws.
Jaw_Heavy heavy jaw Carriers of this gene have large jaws.
Head_Gaunt gaunt head Carriers of this gene have a pinched, gaunt appearance in their face and head.
Hands_Human human hands Carriers of this gene have regular human hands. 0
Hands_Pig trotter hands Carriers of this gene have hands that partially resemble pig trotters. This reduces their ability to manipulate objects. 1 1
ElongatedFingers elongated fingers Long, delicate fingers improve the carrier's manipulation capacity. This aids with many tasks, especially crafting and construction. 1 -1
Headbone_Human human headbone Carriers of this gene have regular human skulls.
Headbone_MiniHorns mini-horns Carriers of this gene grow two small horns protruding from the forehead.
Headbone_CenterHorn center-horn Carriers of this gene grow a single horn protruding from the center of the forehead.
Voice_Human human voice Carriers of this gene have regular human vocal chords.
VoicePig pig voice Carriers have a squealing voice like that of a pig.
VoiceRoar roar voice Carriers have an animal-like roaring voice.
Hemogenic hemogenic Carriers of this gene have a reserve of biological strength powered by a resource called hemogen. The resource can be gained and spent in various ways, all of which are unlocked by other genes.\n\nCarriers lose 2 hemogen per day from biological entropy. 1 1
HemogenDrain hemogen drain Carriers lose an additional 8 hemogen per day from biological entropy. 1 6
FireWeakness tinderskin Carriers have dry, thin skin which burns easily from fire, and their immune systems react very poorly to this kind of threat. Damage from fire is multiplied by 4. 2
FireTerror pyrophobia Carriers of this gene have an intense fear of fire. When fires are close, there is a chance they will have a mental breakdown at any moment. 4
PerfectImmunity perfect immunity Carriers of this gene have archite-enhanced immune systems which intelligently destroy invaders. They are totally immune to most normal illnesses. 3 1
DiseaseFree non-senescent Carriers of this gene do not go through senescence in the normal way. They never get chronic age-related diseases like cancer, bad back, cataracts, or dementia. 3 1
TotalHealing scarless Carriers of this gene have a special type of regenerator cell which can heal old wounds and chronic illnesses like bad back. 4 1
Deathrest deathrest Carriers of this gene must periodically regenerate themselves in a special coma called deathrest. Deathrest takes days, but can confer substantial bonuses. Deathrest can be accelerated and its effects enhanced by the use of a variety of special buildings and technologies.\n\nThose who put off deathresting will suffer from deathrest exhaustion. 6
Ageless ageless Carriers of this gene have archites in the bloodstream which continuously reverse the process of aging. Starting at the age of 13, carriers begin to biologically age slower. By 18, the aging process stops completely. 3 1
Deathless deathless Carriers of this gene have archites in the blood which will sustain their life processes no matter what. As long as the brain remains intact, a carrier of this gene will never die. 7 1
ArchiteMetabolism archite metabolism Carriers of this gene have special archites in their cells that facilitate and optimize metabolism. This improves overall genetic and metabolic quality. 6 6 2
WoundHealing_Slow slow wound healing Carriers of this gene heal from wounds half as fast as normal. 2
WoundHealing_Fast fast wound healing Carriers of this gene heal from wounds twice as fast as normal. -2
WoundHealing_SuperFast superfast wound healing Carriers of this gene heal from wounds four times as fast as normal. -3
Immunity_Weak weak immunity Carriers of this gene gain immunity to diseases more slowly than normal. They may die from infections that others would survive. 2
Immunity_Strong strong immunity Carriers of this gene gain immunity to diseases faster than normal. -1
Immunity_SuperStrong super immunity Carriers of this gene gain immunity to diseases considerably faster than normal. 2 -2
ToxicEnvironmentResistance_Partial partial antitoxic lungs Carriers of this gene are resistant to environmental toxins. They get less toxic buildup from tox gas, polluted terrain, and toxic fallout, but are still vulnerable to direct attacks with venom or injected poison. Additionally, they build up rot stink exposure slower. -1
ToxicEnvironmentResistance_Total total antitoxic lungs Carriers of this gene are immune to environmental toxins, but not from direct toxic attacks. They get no toxic buildup from tox gas, polluted terrain, or toxic fallout, and they are not bothered by acidic smog. They are still vulnerable to direct attacks like venom and injected poison. Additionally, they are immune to rot stink exposure. 2 -3
Sterile sterile Carriers of this gene cannot reproduce by natural means. 1
Fertile fertile Carriers of this gene have a higher chance of becoming pregnant or impregnating others.
Superclotting superclotting Carriers of this gene have extra-power coagulating factors in their blood, and will stop bleeding very quickly when wounded. -1
KindInstinct kind instinct Carriers of this gene are high in trait agreeableness and are very conscientious. They rarely insult others and will sometimes offer kind words to brighten the moods of those around them. They also never judge people by their appearance. -1
ViolenceDisabled violence disabled Carriers of this gene are emotionally and mentally incapable of engaging in violence. They are overwhelmingly resistant to and horrified by the idea of hurting another. 3
Nearsighted nearsighted Carriers of this gene have difficulty seeing at a distance. Their shooting accuracy at long ranges is reduced. 2
StrongStomach strong stomach Carriers of this gene have an extra toxin-filtering organ in their stomach and will never suffer from food poisoning even after eating rotten food. -1
DarkVision dark vision Carriers of this gene see well in low light and are unaffected by mood penalties related to darkness. They have a reflective layer behind the retina that amplifies their ability to see in the dark. -1
KillThirst kill thirst Carriers of this gene lust for the feeling of ending another's life. They will become irritated if they go for too long without killing someone in close combat. 4
FireResistant fire resistant Carriers of this gene have special fast-acting sweat glands and heat-resistant skin. They only take 25% of the normal damage from fire. The chance of them catching on fire is also drastically reduced. -2
Inbred inbred This genetic condition affects a person's fertility, immunity, and mental capacity. -2
RobustDigestion robust digestion Carriers of this gene grow a multi-fold stomach, allowing them to digest raw foods more efficiently than baseline humans. In general, they get the same nutrition from raw food as from if it is cooked. They also don't mind the taste of raw food at all. 2 -2
Instability_Mild mild cell instability Carriers of this gene need less metabolic energy to stay alive, at the cost of reduced stability in their cell-replication machinery. 2
Instability_Major major cell instability Carriers of this gene need much less metabolic energy to stay alive, at the cost of greatly-reduced stability in their cell-replication machinery. 4
PsychicBonding psychic bonding Carriers of this gene have a special neural organ that makes them psychically bond with a lover for life. As long as the lovers are together, they will be happy. If they are physically separated, they will be disturbed by the distance. If one dies, the other's mind will be badly disrupted. 1 -1
PollutionRush pollution stimulus Carriers of this gene get a chemical rush from being exposed to pollution. This makes them move faster and helps them think clearer. A similar gene is found in combat-engineered mega-insects. 1 -1
Unstoppable unstoppable Carriers of this gene are not slowed down when taking damage. 1 -2
NakedSpeed naked speed Carriers of this gene move slower while clothed, and faster while naked. 1 2

Posted

stdout

如何在出远门的时候关闭洗碗机定时启动

家里有个「美的」的某型号洗碗机,一般是积累了当天的碗筷餐具,每天23:00低谷电价之后自动开始定时洗碗。这个功能可以在app上设置实现,还可以设置烘干和通风保存,第二天就能拿出来洁净且干燥的碗筷使用,平时日常这样用起来挺不错。

但是这也导致一个问题,如果节假日出门旅行几天,这个洗碗机里即便没有碗筷,也会傻傻的启动浪费水电,所以需要禁用几天,我的操作流程是:

  1. 掏出手机,解锁屏幕,找到「美的美居」 app并打开
  2. 欣赏该厂家运营设置的精美开屏广告并等待
  3. 打开「场景」
  4. 找到自动化 - 已开启 列表里,关于洗碗机的部分
  5. 找到进入每天自动洗碗的选项,编辑,选择关闭
  6. 保存设置,退出 app
  7. 回到家,又需要按照上面的流程操作一番启用该流程

每次想起这个事,真是又繁琐又蛋痛。特别是回来如果忘记启用,第二天早上起来会发现一篮子未洗的碗筷傻眼。

这次五·一假,我压根忘记了禁用定时洗碗的事。但是回到家添加洗碗粉的时候发现,这几天洗碗机并未启动。问了下是我老婆操作的,她的流程是:

  1. 给洗碗机门板开一条缝

洗碗机不会在门没关严的情况下启动,所以。。。。

我问她这样做是故意的还是不小心的,她说故意不小心的。

Posted

stdout

当前版本gpt的根本弱点

下面的东西仅限当前,不是不可以解决的。

  1. gpt系列数学能力不行,是因为它把每一个数字当成一个整体在背加法表、乘法表。没理解到十进制的精髓

上图来自2020年 GPT-3 论文 22页。主要还是因为 BPE 太糙。

  1. gpt-4能通过一些文字材料和描述感知到每个字母大概长什么样,甚至知道一些简单汉字的结构,但是确切的书写结构它是不知道的。
    所以要避免你的想法被 gpt 感知学会,得考虑每个字的偏旁部首作文章,而且你的变形得很少见。估计这个技巧在将来被AI监视的生活里会被用得上。

  2. 还有一些语音漂移它知道概念但是因为没发音器官所以不能直接感知。但这也不是不能解决,只要语料够大也能摸到规律。相比英语等拼音文字而言,汉字这类表意但是又有同音的文字应该是文本类 AI 的普遍弱点。

如果你要秘密表达一些观点,可以通过描述发音、形状或者动作但是不直接写出来让对方领会。

Posted

stdout

Google 和 Facebook 为什么不行了

Google 最先搞出来 word2vec,最先搞出来transformer 和 BERT,最先搞出来被员工发现 sentient 的 LaMDA,被 ChatGPT 骑脸之后匆忙推出个半成品 Bard

这一切为啥呢?我觉得很可能是 Google的愿景

Google's mission is to organize the world's information and make it universally accessible and useful.

全世界的信息,可以说已经被 Google 成功的「组织」起来了。只不过很可惜,它并没有想要全部「理解」和「消化吸收」,把信息变成「通识」。ChatGPT 正在尝试做到这一点。

微软的愿景是什么呢?

  • "Be what's next." (2010-2012)
  • "Empowering us all"

所以它很快收购了 Skype 推出 Teams,收购 Github 然后投资 OpenAI,结合 Codex (code-davinci-002) 推出 Github Copilot,然后顺理成章搞出来 Office 系列的 Copilot。工具性拉满。

然后说下为啥 Facebook 要 All in metaverse。这个在 2015-06-22 有个 leaked memo 说得很清楚

Our vision is that VR / AR will be the next major computing platform after mobile in about 10 years. It can be even more ubiquitous than mobile – especially once we reach AR – since you can always have it on. It’s more natural than mobile since it uses our normal human visual and gestural systems. It can even be more economical, because once you have a good VR / AR system, you no longer need to buy phones or TV’s or many other physical objects – they can just become apps in a digital store. ... We are vulnerable on mobile to Google and Apple because they make major mobile platforms. We would like a stronger strategic position... We can achieve this only by building both a major platform as well as key apps.
我们的愿景是,在大约十年后,VR / AR将成为继移动设备之后的下一个主要计算平台。它甚至可以比移动设备更普及 - 特别是一旦我们达到AR - 因为您可以随时随地使用它。它比移动设备更自然,因为它使用我们正常的人类视觉和手势系统。它甚至可以更经济,因为一旦您拥有了一个好的VR / AR系统,您就不再需要购买手机、电视或许多其他物理物品 - 它们只能成为数字商店中的应用程序。我们在移动设备上对谷歌和苹果公司非常脆弱,因为它们制作了主要的移动平台。我们希望拥有更强大的战略地位... 我们只有通过建立一个主要平台和关键应用程序才能实现这一点。

问题是,VR/AR 并不是手机的下一代。

Posted

stdout

Play Cities:Skylines on Windows/MacOS without Paradox Launcher

Haven't played Cities: Skylines for ages, so I re-installed it on Steam, after some updates there's this new Paradox Launcher which serves no purpose but wasting time. So I decided to remove it, googled some guide and unfortunately it contains too many steps and some error occured while following it. Here's the simpler & correct step of what I did instead.

After the game is downloaded, open the Steam library, find the game, right click, choose Properties...

Next for the Launch Options, type

on Windows

cmd /c start cities.exe %command%

on MacOS

/Users/$USER/Library/Application\ Support/Steam/steamapps/common/Cities_Skylines/Cities.app/Contents/MacOS/Cities %command%

Make sure you replace $USER with your username

Then the game works like good o' times, straight to the menu.

Posted

stdout

《阿凡达2:水之道》是一部辱日片啊

昨天刚看完,整体感觉画面还是美轮美奂的。我没看出来有什么故弄玄虚的情节,也没觉得网上说那些故事性不行到底不行在哪里。不过有很多场面感觉不是怎么适合带娃去看,好多暴力、流血、残忍包括几十秒展示捏碎一个真正的骷髅等镜头。

在其中一个 Tulkun Hunting 大场面,一开始还对为啥找个东亚面孔扮演反派感到奇怪

avatar2 whaling crew

然后就出现了好几秒的「日捕」爆炸捕鲸叉的特写。这是把「日本捕鯨協会」写脑门上了啊。。。

日捕

截图来自tiktok不是很清晰。但是这妥妥辱日了吧。。。说阿凡达2不好看的,是不是拿了日本外务省的钱?233333

看完网上搜了下讨论,有人说这是 日浦 Hiura 的名字,但是这又不是一个常见的日本姓名 ,所以我觉得 James Cameron 这故意用个错别字避免刚正面。

现实中的日本捕鲸叉长这样

japanese whale harpoon

十分写实了。而且捕鲸一开始也是为了鱼油,而且日本捕鲸也是冒充科研用途,和电影里的情节不能说雷同只能说。。。一模一样?

Posted

stdout

文字的毒害

看到个号称不支持 css/js/wasm 和图片的浏览器 Kristall,觉得这玩意太蠢了。这让我
想起柏拉图的 Phaedrus 里讲了苏格拉底的一个段子,苏格拉底作为欧亚大陆的头号智慧导师,博学多才,却没留下传世著作。因为但是他老人家本身是反对文字和书写的,有这么几个原因:

第一条,文字伤害记忆和智慧。Letter is an aid not to memory, but to reminiscence, and you give your disciples not truth, but only the semblance of truth

首先值得注意的是,这里很有可能指的是拼音文字里的「字母」 而不是古埃及神 Theuth 创作的象形文字。我认为发音字母构成的书写体系肯定是对记忆有害的,文字上的词性和变位更是对视力和理解力的低效率的折磨。快速理解都会强调 Subvocalization 的重要性。汉字作为一个 ideogram 真是一个神发明。总体来说,这条道理是对的。文字只是记忆的粗略模拟和辅助,是真理的外表轮廓。

知识其实这玩意可以被大致分为两类:propositional knowledge 和 know-how。命题和过程知识。

什么是命题?比如「一只青蛙四条腿」就是典型的命题。人们可以根据生活经验获得大量命题;

过程知识就比较微妙了。人们可以通过文字获得过程知识,比如你可以通过学习五线谱了解乐谱如何和发音一一对应,观看钢琴视频得知乐器如何演奏,但是让你实际上手呢?依然不会。

文字就容易造成一个错觉,把结论和过程的「描述」记住了,不等于你获得了真实的知识。有的时候甚至你记下来的就是个结论,连它内在逻辑都无法知晓。孔老夫子就说过「学而不思则罔,思而不学则殆」。文字是有欺骗性的。很多时候你写下来了背下来了以为就记住了。实际上你只记住了一个表皮。

第二条,文字是固化的,死气沉沉的,不如口头语言丰富有活力,不如口传心授有感染力。这个理由,也有局限性。因为文字表达能力在苏格拉底的时候不行,但是这玩意可以进化的。几千年演化下来,文字的表达能力一点也不差。

第三条,文字使语言失控。苏格拉底的原话是这么说的:“一件事一旦被写下来,不论什么内容,到处都会流传的,既会传到能看懂的人的手里,同样也会传到无关的人手里。文字本身并不知道如何与正直的人说话,也不知道如何不与邪恶的人说话。因为文字没有自卫或自救的能力。” 这个道理其实非常值得深思。

书写不是真理的后代,而是真理的私生子。人们歪曲理解文字,就如同私生子受到欺负没有亲生父母帮忙一样。苏格拉底还说,智慧的语言是用来雕刻学习者的心智的。需要知道什么说什么话,什么时候闭嘴。知识是活的,有多面性,需要对听众定制诉说,而不是像一副绘画那样只从一个角度说明问题。理智的人说话就像在花园里埋下一颗种子,听者心里才会开花结果。而不是一味挖洞翻土。

道理讲了这么多,其实对我的启发就是传播思想。喊死板口号是绝对没有用的。必须得定制化、个性化传播。如同现在推荐系统一样,做到千人千面。真正厉害的营销和洗脑不是「羊羊羊」那样反复重复洗脑,而是静心巧妙的植入。

想到这里,我觉得,以前 PC 时代发布视频,是由少数精英团队的制作,通过 Flash/MPEG这种录制成视听作品或者节目,扩散式的单向散开传播;3G 4G 带来的高带宽普及,智能机带来的H.264专用处理芯片的普及,让人人拍视频成为了可能,让全球迎来了一个视频传播思想的时代。大家看不起的营销号洗稿行为,在我看来是一个极其合理必然。知识的传播,本来就应该自然发生变异,我们只是缺乏一个有效的优胜劣汰的定向过滤器。把一件事前因后果用文字写清楚的人不多,但是文盲都可以拿起手机录制一段讲话。所以在人人都能发视频的时代,打破了文字的垄断,带来大量的土味和「俗」味视频。因为媒介的改变,不仅是从文字到图片、视频的升级那么简单,而是人和人对话、交流、碰撞的一次彻底颠覆;以前的大道理、小道理是基于文字的浓缩,抽象成一个又一个的符号,比如国家,比如民族,比如儒家、清教徒这种门派。现在抛弃抽象,回归到具象。在文字的时代,搜索引擎为王。因为这种技术有一定门槛,而且当时互联网上全是「超文本」的海洋。高校的理论知识是网页,门户的新闻是大段文字,个体用户参与的 BBS 讨论也是文字。现在不一样了,学术报告是扫描版pdf,门户都是app里的walled-gargen,群众的议论全是长截图。文字沦为配角,作为字幕存在于某一帧配图的下方,或者是一个箭头指向的注解。最具冲击力的,往往是影像,比如唐山打人的监控 footage,再比如导致 BLM 的那张 Floyd 跪脖子的图片。

社会现代化意味着传统、稳定的社区纽带被打散,现代人的生活成为一场漫长的“流浪”,人们不断被抛入陌生的地点、陌生的群体、陌生的关系,“独自承受”一切挑战。这种原子化状态驱动着弗洛姆所说的“逃避自由”心理——人们试图在不断分崩离析的世界中抓住一点什么,哪怕抓住的仅仅是“想象的共同体”。

从这一点上来说,我们回到了千百年前那个知识以口口相传的时代。面对这样的历史巨变,我个人的见解是:

  1. 口头交流比任何一个时代都更加重要。中小学应该压缩文字书写的时间,改为制作 ppt 宣讲技能培训和口头辩论。
  2. 应该把音韵、对联、单押、双押、rap、music、的重要性提高。现在自媒体 BGM 水平太低,而且说服力和韵脚好坏成正比。
  3. ChatGPT 是一把双刃剑,既可以在一段文字里无缝植入商品广告,也可以解决苏格拉底反对 letters 提出的文字的种种缺点,把文字变成个性化、启发式学习一个利器。
  4. 下一代说服力工具,应该是类似 ChatGPT 但是作为虚拟形象对现实生活的再加工和演绎。技术侧需要有对图片和视频「表达的思想」的理解能力。

Posted

stdout

Jira里JQL实现查找上一个「工作日」的任务

厂里终于开始用笨拙的Jira了。总体用下来跟 Teambition 相比各有各的笨处。。。

默认的 board 过滤器有点不好用。本着为了stand-up meeting和周报快速检索上一个工作日和当周干了什么,找到一个 filter 写法:

JQL过滤上一个工作日干了啥:

(updatedDate < endofweek("-8d") AND updatedDate > startofday("-3d")) OR updatedDate > startofday("-1d")

这里不是简单的查昨天干了啥。比如周一需要看周五干了啥。所以有个 -8d-3d 的神奇操作。具体的逻辑自己琢磨

过滤本周干了啥

updatedDate >= startOfWeek(0)

Jira这种资本主义软件是没法适配我们社会主义法定节假日的设定。所以遇到非双休放假还是判断起来比较蛋痛。

其实我比较反感每天早上开站会。我比较推荐每天下班前最后10分钟开会。好处有:

  1. 当天干了什么记得很清楚。明天需要干什么也很显然。
  2. 如果谁拖堂,谁就承担对不起大家不能准时下班的道德压力
  3. 如果都开会了你事情都还没做完,要么你事情安排得太多,要么你完成能力不足,要么你时间安排有问题。

之前在两个公司这样执行过。结果被大老板强行改到早上开会了。无语。

Posted

stdout

Web Animation实现页面逐渐变灰

在2006年我开始网上留下印记的时候,写了一个 dHTML 特效模拟XP关机对话框变灰, 那个时候大行其道的还是IE6,用的还是微软特有的 filter:progid:DXImageTransform.Microsoft.Fade 叠加两个<div>实现。

今天(2022-11-30)恰好长者去逝,学习一下最新的css姿势再实现一次。本来想用 CSS Animation+Filter实现,还是比较麻烦。比如我参考的例子是模仿 Red Dead Redemption 2 照片效果css

  #invert{
    filter: invert(1.0) grayscale(1.0);
  }
  #photo{
    animation: blur 5s infinite alternate;
  }
  @keyframes blur{
    from{
      filter: blur(3px);
      opacity: 0;
    }
    to{
      filter: blur(0px);
      opacity: 1;
    }
  }

这个方式有个问题就是需要单独设置一个 <script> 元素。后来经过 @yuanchuan23 的指点发现可以通过 Web Animation 更简单的实现:

  document.body.animate([
    {filter: 'brightness(none) grayscale(0%)'},
    {filter: 'brightness(10) grayscale(10%)', offset: 0.2},
    {filter: 'brightness(1) grayscale(100%)'}
  ], {duration: 2000, fill: 'forwards'})

这样写比 css 方便多了。加一个类似照相机闪光灯过曝的效果。其中 fill: 'forwards' 可以让变灰之后停下来。

放一个 demo 在这里:https://lab.est.im/grayscale_animation 点一下可以再变一次

Posted

stdout

8行javascript录制屏幕

看到个骚操作,玩了一下,写了个 bookmarklet

navigator.mediaDevices.getDisplayMedia({video:true}).then(function(stream){
  const mr=new MediaRecorder(stream,{mimeType:"video/webm;codecs=h264,opus"});
  mr.ondataavailable=function(ev){
    const a=document.createElement('a');a.href=URL.createObjectURL(ev.data);
    a.download ='screen_capture.webm';a.click();
  }; // will be called when .stop() with ev.data as a Blob
  mr.start();
})

这段代码也是很多WebRTC共享白板的实现原理。看起来也很简单,而且不需要额外的权限确认。

如果不需要另存为一个名字,则可以直接 location.assign() 又节约几个字。不知道还有没有办法继续节约代码量?

有点很好奇为啥chrome不支持 .mp4 封装。偏袒自家的 webm ?

Posted

stdout

JS播放 DTMF

玩 WebRTC 的时候发现它居然内置支持了 DTMF 拨号音

这里就简单科普一下,DTMF 就是手机电话拨号音,数字按键按下去嘟嘟嘟响那个。具体就2个频率调谐:

DTMF 1209Hz 1336Hz 1477Hz 1633Hz
697Hz 1 2 3 A
770Hz 4 5 6 B
852Hz 7 8 9 C
941Hz * 0 # D

于是兴趣来了想做一个 demo 玩玩,写了半天调试不通,报错

VM987:1 Uncaught DOMException: Failed to execute 'insertDTMF' on 'RTCDTMFSender': The 'canInsertDTMF' attribute is false: this sender cannot send DTMF.

仔细研究了下这玩意,原来 WebRTC 不直接传音频,而是发一个控制信号,由接收方播放出音频,所以你需要本地创建一个真实的双工 RTC 并打开麦克风权限,才能播放DTMF音频。

WebRTC doesn't send DTMF codes as audio data. Instead, they're sent out-of-band, as RTP payloads. Note, however, that although it's possible to send DTMF using WebRTC, there is currently no way to detect or receive incoming DTMF. WebRTC currently ignores these payloads;

这样就走远了。于是又手痒用 AudioContext() 糊了一个。一共不到1000字节。

https://lab.est.im/dtmf 源码

嗯。就挺简单的。特别是那个 oscillator有个 .stop(0.2) 参数可以避免 setTimeout 的麻烦。网上很多教程整复杂了。

btw 把一些老脚本挂在 lab 域名下。
btw2 把 RSS 改成全文输出了。

Posted

stdout

Cloudflare worker 返回浏览器ip:端口

看到个贴 返回客户端连接用的 IP 和端口号,想起来也需要熟悉一下 Cloudflare worker,于是拿来练练手

  export default {
    async fetch(request, env) {
      return await handleRequest(request).catch(
        (err) => new Response(err.stack, { status: 500 })
      )
    }
  }

  async function handleRequest(request) {
    const { pathname } = new URL(request.url);

    if (pathname === "/ip") {
      const ip = request.headers.get('CF-Connecting-IP');
      return new Response(ip + '\n');
    }

    if (pathname === "/ip:port") {
      const ip = request.headers.get('CF-Connecting-IP');
      const port = request.headers.get('est-Connecting-Port') || '?'
      return new Response(ip + ':' + port + '\n');
    }

    return new Response('Tools:\n /ip\n/ip:port\n');
  }

遇到一个问题,Cloudflare 官方有提供浏览器客户端IP变量 CF-Connecting-IP 但是缺少端口,查了下可以自己通过 cf.edge.client_port 这个变量获得, 方法是在进入 CF网站管理 面板 → Rules → Transform Rules → HTTP Request Header Modification 里添加一条请求头规则。如图:

但是 header 不能以 CF- 开头,所以这里改成 est-Connecting-Port,就能获得客户端请求的 IP+端口 了

测试地址: t.本网站根域名/ip:port

目前还缺:IPv6 的正确表示。。。。

Posted

stdout

macOS动态定时弹提醒

上班地方有个付费计时,想在倒计时结束前在电脑弹个提醒,手动结束计费,趴下来API了。接下来就是如何弹窗了。

本来想用macOS自带的 UNUserNotificationCenter 通过 PyObjC 撸个弹窗,遇到两个问题:

  1. 系统自带的 PyObjC 太老了。还是Python2.7的。在/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
  2. 只能兼容旧款的 NSUserNotification,状态为 macOS 10.8–11.0 Deprecated
    新的API调用需要进程签名。
    codesign -dv --verbose=4 /usr/bin/python2.7

只有官方的python2.7签名了 anaconda/miniconda 的都没签名。算了。还是直接 AppleScript 走起

  osascript  -e 'display notification "还有1分钟到达时辰" with title "准时付费" sound name "Frog"'

想在这个提示里加一个 snooze 1分钟功能,但是好像无法支持。罢了。2333

Posted

stdout

终于吧 blog 的自动构建 pipeline 配置好了

目前博客是基于 pelican 的 static-site generator,一个 python 模块本地渲染成静态文件然后通过 github pages 发布。

源码放在 bitbucket,之前比较懒都是本地跑,一直想改成 CI/CD pipeline,今天终于配置好了。记录一些坑吧

  1. bitbucket pipeline 不同 step 之间数据是不共享的。本来想用 parallel 功能在下载 pip 模块和 git clone 皮肤做个并行,后来发现 artifact 拷贝速度还不如串行去跑。就所有操作一个 step 做完
  2. 博客源码和生成结果可以直接同步提交信息:

    msg=$(git log -1  --pretty=%B)
    cd output
    git config user.name est
    git config user.email 邮箱
    git add .
    git commit -m "$msg"
    git push
    
    1. 由于 pip 模块装了不知道在哪里,所以调试 pip 安装路径方法:

      python -c 'import site; print(*site.getsitepackages(), sep="\n")'

  3. pelican 默认居然没支持 Markdown 也是醉了

    python -m pip install pelican==4.8.0 Markdown
    
    1. 自定义皮肤依赖,由于 submodule 我没玩转,所以这样安装:

      git clone --depth=1 git@github.com:est/aether-pelican.git || (cd aether-pelican; git pull --rebase)

Bitbucket 每个月有50分钟的免费pipeline 时间,自己用下来差不多29秒 build & publish 一次。感觉每个月也写不了60篇,所以目前是够了。

接下来的优化:

  1. 新图片通过 squoosh 压缩优化
  2. 通过 cloudflare worker 替换掉 disqus 评论功能

Posted

stdout

袁亚湘院士数学不错,但是历史不行

网上看到一个院士的段子 1 2

搜了下,2020年3月14日,在中国工业与应用数学学会举办的网络科普活动中,中国科学院院士袁亚湘带来题为“数学漫谈”的主题报告

https://mp.weixin.qq.com/s/JllGeuveomjFDFg3C5r9FQ

关于π有很多神奇的故事。因为时间关系,我今天就讲其中的一个。不知道在座的观众有没有看过一个美剧叫做《疑犯追踪》。在第2季11集中,Harold Finch说过在π里面能够找到任意数列。当然这个东西数学家并没有证明,但是很多数学家认为可能是真的。
我试了试,我挑了我们建党的日子19210701,出现在π的第44,842,733位。如果输入去中华人民共和国成立之日,也就是19491001的话,就要往后很多,要到82,267,377位。开个玩笑,π都说明了要先有共产党才有新中国。
一个网友要求我要查一下1314,大家看到1314到3902位就找到了。所以各位观众,1314(一生一世)不是那么难啊。但是如果1314前面加个520(我爱你)的话,那就难多了。如果你要找5201314,那就要查到200多万位。所以同学们一定要记住啊,实际上爱情还是很艰难的。

这段话数学上来说是没问题的,但是院士可能没注意历史,建党的真正时间是一大正式召开的时间,1921年的7月23日

那么我们把两个时间再去看一下在 Pi 里的位置:

https://www.angio.net/pi/

  The string 19491001 occurs at position 82267377
  The string 19210723 occurs at position 101114100

所以这说明了啥?

Posted

stdout

Add UTF-8 BOM header to Flask-Admin export

A simple trick to add additional bytes prepending Flask's stream_with_context

  from flask_admin.contrib.sqla import ModelView
  from itertools import chain

  class MyAdminView(ModelView):
      can_export = True
      column_export_list = ['field1', 'field2']

      def _export_csv(self, return_url):
          r = super(MyAdminView, self)._export_csv(return_url)
          r.response = chain((b'\xef\xbb\xbf',), r.response)
          return r

I am still using Python2 and Flask-admin shit. It's a shame but anyway.

Posted

stdout

牛津树/Good English/典范英语里的宗教私货

牛津树 Oxford Reading Tree 是被国内咪蒙妈妈们吹得油爆爆的来自英国的少儿学习体系,Good English/典范英语 是中青出版社的部分摘录英语教材。

偶然发现这个 牛津树【4-29】Mosque School,讲了一个叫 Adam 的学龄前儿童机缘巧合进了经文学校的短故事。甚至如果这是从古兰经某个传教段子改编出来的我一点都不惊讶。

引起我注意的是小白帽的装束,以及牌子那几个字 Masjid-E-Noorul Islam

费了点心思搜了下,也拼为 Nurul IslamNur ul-Islam,意思是 light of Islam 伊斯兰之光

国内很多中专职高出来的英语老师可能没理解这里面的内涵。什么叫潜移默化植入意识形态啊?!当人们觉得这是文化「多样性」很正常?!以为清真学校就是普通的讲宗教的学校,其实经文学校传授的东西非常不简单。摘抄一些百科:

经文学校分为“地尼买克塔普”、“买德里斯”、“卡里哈纳”三种。
经文小学是中国伊斯兰教经堂教育建制之一。俗称“经学”。专为年满6~7岁的穆斯林儿童所设,多在清真寺内,少数系私人开办,在教师或师娘的住宅内。设在清真寺内的经文小学,教师皆由该寺教职人员担任。课程由认识、诵读、书写阿拉伯字母开始。待学会拼音之后便相继以《索尔夫》、《穆阿济》、《扎佳尼》、《阿瓦米勒》与《米苏巴哈》为教科书(俗称“连五本”)。进而开讲《亥听》(即《古兰经》选读本)、《古兰经》。经文小学的学童至10岁左右,率多辍学从事家庭生产劳动或参加社会职业,只有少数人转入经文大学,继续接受高一级的伊斯兰教经堂教育,被称为“满拉”或“海里凡”。经文小学没有严格的学籍管理,招生不定期,也不经过考试,辍学或退学甚为自由,没有班级排列。学童的进度不齐,故普遍采取复式教学法。另有伊斯兰教常识课程,每日放学前集体朗诵(系中国阿訇自编启蒙教材,俗称“杂学”)。“杂学”的内容包括沐浴、礼拜、斋戒、丧葬及游坟等宗教活动应念的都阿、苏赖。1911年辛亥革命成功后,一些设于清真寺内的经文小学开始聘请懂汉文的穆斯林给学童加授汉文启蒙性教科书。女童上经文小学率多在师娘私宅。学费没有定制,学童家长可量力向师傅(师娘)不定期地馈送海迪耶。

官方介绍可以参考 https://cati.nwupl.edu.cn/dtzx/hswx/10307.htm 中 「保守教派的兴起」 这一部分,维基百科 扎吉德 Jadid民间经文学校初步观察

如果你觉得这样的东西堂而皇之放在英语教材里没啥问题。。。我也没啥好说的。。。

其中这一段比较有趣:

情况与当时还属于苏联的中亚五国是一致的。这个过程中,瓦哈比派(Wahhabism)因为主张“回到《古兰经》中去”,注重个人对经典的领悟而非学者的解释,门槛很低,容易接受

这不王阳明心学么。。。。释经权的下放和离散化。。。这后果。。啧啧。。。。参考之前一篇《中世纪中国的宗教改革

Posted

stdout

徐州丰县,以及孝文化

抬一张地图出来,

鲁苏豫皖四省,如果让你选一个最乱的地,你可能选哪里呢?

说起来,徐州对我的印象,只是《三国演义》上一个模糊的地理概念,第一次稍微立体的感受徐州,还是在《睡前消息29 · 没有“淮海省”,徐州还是有当“省会”的命?》上看到,徐州是一个距离周边大城市都非常远的人口又比较密集的城市。

后来,从罗辑思维·启发俱乐部·第26期:显性力量越大,隐性麻烦越多》上扒到徐州周边刷过的历史人物,就更印象深刻了

这里是砀山,刘邦拔剑斩蛇起义的地方,它属于安徽省,但它同时被另外三个省包围,北部是山东,东部是江苏,西部是河南。
从这个地方向东北,开车不到一个小时,就能到达江苏省的丰县。刘邦就是丰县人。辅佐刘邦建立汉朝的很多人,像萧何、卢绾、王陵也是丰县人。
丰县的东边是沛县,刘邦集团的樊哙、曹参、周勃都是沛县人。你可能会说,刘邦搞事业,拉着乡亲入伙很正常啊。
从丰县县城向东南出发,一个半小时车程就会到达徐州。在秦代,这里叫彭城,项羽建立西楚,都城就是彭城。奇怪了,为什么项羽的都城离刘邦家这么近呢?
继续向东南驱车一个半小时,会到达今天的宿迁市,在秦代,这里叫下相,项羽的故乡。你看,连项羽家距离刘邦家好像都不是很远。
然后,从项羽的家乡向西出发,开车一个多小时,就会到达安徽省灵璧县,“霸王别姬”、“乌江自刎”的故事就发生在这附近。
踩着油门朝西北走,不到一个小时,我们来到安徽省宿州市。在秦代,这里沼泽遍地,名叫大泽乡。没错,陈涉吴广起义,就发生在这里。
是不是觉得有一个问题已经呼之欲出了?秦朝灭国的可是六国啊。齐楚燕赵魏韩,那么大一片地方啊。但是怎么在秦朝末年找麻烦的就集中在这么一小块地方呢?那么大片的地方不出事,偏偏就这里出事?
对,佐竹靖彦给这一小块地方起了个名字:“泗水系月牙形水乡山泽地带”。
在那个时代,这个地方是一片月牙形地带,泗水河川行而过,地势低、潮湿,形成了大量的水乡山泽。在秦代,这里有个地方叫大野泽。听这个名字,你就能感受到三个特点:水多、荒凉、面积大。后来随着大野泽不断缩小并且位移,到五代的时候,大野泽变成了梁山泊。
一听到梁山泊,你就明白这个地方的意义了。官兵到不了嘛。

另外,徐州还是汉帝国的掘墓者曹操起家之处,唐帝国覆灭者黄巢朱温老巢,巨盗宋江也就是在河北、河南、山东、江苏交接的地方来回变动

然后结合时事,为什么这里拐卖人口多?

很多摸鱼群的分析是孝文化,传宗接代的思想在作怪,什么养儿防老,养儿挣钱,但读到2020年这篇Yixi演讲的文章,我就不淡定了

吴心越:养老院调查真相,让老人恐惧,问题严重性超出你的想象!

小时候,父母对孩子通常都是很强势的。但是随着年龄增长,你会发现父母对你越来越弱势,其实就是这个原因。养老院里那些无儿无女的老人,其实和学校里面的“孤儿”没有本质的区别,都属于被欺负的对象。在学校,父母是孩子的依靠。在养老院,子女其实就是老人的依靠。
哪怕有些子女很混蛋,很不负责,但他是维权的直接受益人。也许子女没有孝顺的动力,但是却有替父母维权的动力。越是混蛋的子女,战斗力就越强,父母出了问题,他们能把养老院讹到破产!
就拿护工来说,知道你有哮喘,偏要把你安排在通风不好的地方;知道你和谁有矛盾,偏要把你俩安排在一个房间;知道你腿脚不利索,偏要把你安排在楼层较高的房间;故意把你安排给有传染病或者大小便失禁的老人一个房间.....这种情况,尤其是在养老行业,从来就不是个案,而是普遍性的现象。
如果找院方投诉,就相当于把护工得罪死了,只要护工不被开除,如果不换养老院,剩下的日子,老人的日子只会愈来愈难过。
如果老人在外面有子女,起码护工还会有所忌惮,知道他自己做的太过分,会有人来闹,就会有所收敛。
这个时候,老年人有子女的最大意义,就是出了问题,有人帮你喊冤,哪怕他仅仅是为了搞一笔钱。

所以现在突然明白,为啥乡土中国,熟人社会,有拼命生娃的做法,讲究「孝」道了。这TM的是生存的最低保障,战略威慑能力啊。。。

如果说大型中心省会城市代表秩序和文明,那么徐州周边的几百公里没有大城市的平原村落里,就是社会正义能延生到的末端,这个时候,基层秩序只能由血缘绑定。即便是贩卖人口交换来的亲人。

Posted

stdout

笑话了千年刻舟求剑,水面搜救的图案却叫Victor Sierra

今天的第一个收获,Victor Sierra这个搜救图案

来自 Why This Zig-Zag Coast Guard Search Pattern is Actually Genius - Smarter Every Day 268,是美国 coast guard 标准化的水面搜救路线之一。这种图案覆盖的面最大并且效率最高,而且最重要的一点,考虑到水是流动的,能够跟随一起漂流的相对速度去找到 PIW (person in water)

于是我就想起大家都熟悉的成语——「刻舟求剑」。这是2000多年前一个河南贩子吕不韦编的一个讥讽九头鸟的段子,由于这个成语太熟悉了,我甚至开始怀疑课本上是不是掐头去尾没讲完,跟之前那个「守株待兔」恶毒的政治隐喻一样,于是去翻了下原文。好家伙,不能说雷同,简直就是一模一样。它出自《吕氏春秋·察今

察今一共三段,第一段说,按着先人制度惯性因循守旧的人都是大沙雕;第二段说,荆州人拿了过期的水文资料强行过河结果沙币了。第三段说,好剑要的是锋利,不必是莫邪的名称;好马要的是能行千里,而不是骥骜的名称。成就功名的人,就是老板的好工具人。然后就抬出来了著名的讽刺湖北佬的段子:

楚人有涉江者,其剑自舟中坠于水,遽契其舟, 曰:「是吾剑之所从坠。」舟止,从其所契者入水求之。 舟已行矣,而剑不行,求剑若此,不亦惑乎? 以此故法为其国与此同。 时已徙矣,而法不徙,以此为治,岂不难哉? 有过于江上者,见人方引婴儿而欲投之江中, 婴儿啼,人问其故,曰:「此其父善游。」 其父虽善游,其子岂遽善游哉? 此任物亦必悖矣。荆国之为政,有似于此。

第二个收获就是,果不其然又是一个阴阳怪气讽刺朝政的段子。看来每一个成语背后都有一个键政的动机哎。

第三个收获,没想到刻舟求剑之后,还有个「其父善游」的更离谱段子卧槽。。。

然后突然就想到了近期第四个收获,王中林院士拓展麦克斯韦方程组,在渣乎上看的一路下来,无论是懂哥,还是路人,几乎全是在喷。但是恰好就在 HN 有人投递了一则如何通过伽利略旋转和洛伦兹旋转认识特种相对论的线串

https://twitter.com/MarkusDeserno/status/1482811504424542211

其中从 arctan(m12) = arctan(m1) + arctan(m2) 推到 tan(x+y) = [tan(x) + tan(y)] / [1 – tan(x)·tan(y)]

再到 m12 = [m1 + m2] / [1 – m1·m2]

好吧,我也假装我看懂了!

最后一个感触,大家都顾着去看2500年前楚国人和当今院士的笑话了,却很少有人正面分析一下问题。

Posted

stdout

python-phoenixdb 设置读取超时

PhoenixDB 居然是基于 requests + pb2 做的传输层协议。连接的是 queryserver 的 http 接口

avatica/client.py 源码 可以得知这玩意默认是不支持设置read timeout的。强行hack一个:

import phoenixdb
import functools
c = phoenixdb.connect('http://localhost:8765/', autocommit=True, auth="SPNEGO")
c._client.session.request = functools.partial(c._client.session.request, timeout=2)

partial 这个神奇的魔法可以设置一个默认参数。其它地方传入了别的timeout也可以自行覆盖,无缝兼容。

Posted

stdout

Microsoft 官网的 403 错误

$ curl -H "User-Agent: User-Agent: Mozilla/5" -kvs 'http://www.microsoft.com'
*   Trying 118.123.102.107...
* TCP_NODELAY set
* Connected to www.microsoft.com (118.123.102.107) port 80 (#0)
> GET / HTTP/1.1
> Host: www.microsoft.com
> Accept: */*
> User-Agent: User-Agent: Mozilla/5
>
< HTTP/1.1 403 Forbidden
< Server: GHost
< Content-Length: 306
< Content-Type: text/html
< Mime-Version: 1.0
< Expires: Wed, 29 Dec 2021 14:42:28 GMT
< Cache-Control: max-age=0, no-cache
< Pragma: no-cache
< Date: Wed, 29 Dec 2021 14:42:28 GMT
< Connection: keep-alive
<
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;mscom&#46;errorpage&#46;failover&#46;com&#47;403&#47;403&#46;htm&#63;" on this server.<P>
Reference&#32;&#35;18&#46;1c697b76&#46;1640788948&#46;3b971c1f
</BODY>
</HTML>
* Connection #0 to host www.microsoft.com left intact
* Closing connection 0

不知道为何会指向 http://mscom.errorpage.failover.com 这域名也不是微软的。

Posted

stdout

Chrome 阉割指南

不折腾不舒服。继之前 Edge调教 之后记录一下 Chrome V70怎么阉割。

下载:

win 32 | win 64 | Mac

上面的链接已经年久失修了,对比了一下获取最新版本的脚本发现 Google 已经不提供老链接了。

curl -vsk 'https://tools.google.com/service/update2' -HContent-Type:text/xml --data-binary "<?xml version='1.0' encoding='UTF-8'?>
<request protocol='3.0' sessionid='{est-blog}' installsource='ondemandcheckforupdate' dedup='cr'>
<os platform='mac' version='66.6.6666.66' arch='x64'/>
<app appid='com.google.Chrome' ap='' version='' nextversion='' lang='' brand='GGLS' client=''><updatecheck/></app>
</request>"

安装完毕之后

cd '/Applications/Google Chrome.app/Contents'
touch Frameworks
sudo chown root:wheel Frameworks
sudo chmod 000 Frameworks

cd ~/Library/Google
rm -rf GoogleSoftwareUpdate
touch GoogleSoftwareUpdate
sudo chown root:wheel GoogleSoftwareUpdate
sudo chmod 000 GoogleSoftwareUpdate

Posted

stdout

认识「洋节」

都说 xmas eve 是洋节,我们就来数一数到底有哪些洋节

  1. 元旦1月1日。由北「洋」军阀袁世凯设立。依据是内务部总长朱启钤《定四季节假呈》,把原来农历(夏历)的正月初一的「元旦」挪用到了格里历一月一日。格里历(Gregorian calendar)引入中国是1912,元旦正式生效是1914年。
  2. 春节。严格的来说,传统春节是二十四节气中的“立春”,是太阳历。后来被袁世凯直接拍板成农历正月初一。而夏历正月初一一般被认为是最接近立春之朔日(月缺之日)。农历最后一次农历大改版为《授时历》,发生在崇祯二年(1629年)九月,由礼部左侍郎徐光启发起,编撰者:耶稣会的龙华民(西西里人)、罗雅谷(米兰人)、邓玉函(瑞士人)、汤若望(日耳曼人)。
  3. 生肖其实不跟农历绑定,而是跟 二十四节气绑定。二十四节气最后一次改版,也是上面那个《时宪历》那帮洋传教士定下来的。有意思的是,春分那一天恰好开始白羊座,其实我最近才知道二十四节气就约等于12星座,也就是黄道十二宫。如果要说农历《时宪历》含洋量太高,那么他之前的《授时历》编撰者是河北人郭守敬,生于元太宗(窝阔台)三年(1231年)于邢州,由他祖父郭荣(金国人)抚养成人,说起来这老兄甚至顺着蒙古人统治下的锡伯利亚去过北极科考,同一位尼泊尔建筑师阿尼哥合作搞过天文台,十分了得。
  4. 妇女节,全称为「“三八”国际劳动妇女节」。1857年3月8日,美国纽约的制衣和纺织女工走上街头,抗议恶劣的工作条件和低薪。尽管后来当局出动警察攻击并驱散了抗议人群,但这次抗议活动促成了两年后的3月第一个工会组织的建立。1917年3月8日,在俄罗斯帝国首都彼得格勒,纺织女工举行罢工及游行,遍及整个城市,也导致俄罗斯革命的开始。
  5. 清明节,二十四节气之一。太阳到达黄经15°
  6. 劳动节,1884年,劳动骑士团 Knights of Labor 倡导「8小时工作制」;10月,行会与工会联盟召开大会确定1886年5月1日为要求八小时工作制成为标准的日期,美国工会准备举行总罢工支援;1886年5月1日,数千名参加罢工和参加全美各地举行的集会的工人唱起了《八小时》(Eight Hour)一歌;接下来发生干草市场屠杀(Haymarket affair)。
  7. 端午节,据传是为了纪念恨国诗人屈原,来自「蛮夷」楚国
  8. 儿童节。1949年11月国际民主妇女联合会在苏联莫斯科所召开的执行委员会议,会上为纪念悼念1942年捷克利迪策村对儿童的屠杀及伤害,以及全世界所有在法西斯侵略战争中死难的儿童,为保障世界各国儿童的生存权、保健权和受教育权及改善儿童的生活,决议以6月1日为国际儿童节。1949年12月23中央人民政府政务院第十二次政务会议于日通过了《全国年节及纪念日放假办法》,其中规定6月1日为儿童节。
  9. 建军节,由 契丹苏维埃共和国 (Kitajskaja Sovetskaja Respublika, Кита́йская Сове́тская Респу́блика) 于1933年6月决定,将8月1日定为中国工农红军的建军节。
  10. 圣纪节,穆罕默德的诞辰与逝世恰巧都在伊斯兰教历三月十二日,回回三大节之一。

再说一下真正被 appropriation 的「洋节」

  1. 圣诞节。异教徒的节日,四舍五入冬至日,古埃及太阳神 Ra 的节日,被(东)罗马帝国拿来宣传成中东人耶稣的生日。。。摊手。。。比如犹太人就不过圣诞节,但是他们在这一天吃中餐
  2. 母亲节,由12岁女孩安娜·贾维斯(Anna Jarvis)发起,被商家利用,安娜用下半辈子反对母亲节
  3. 父亲节:抄袭母亲节。

Posted

stdout

如何可视化跟踪分析 Python import 耗时

首先在本地安装 pip install tuna

然后跑个分,需要 CPython >= 3.7,输入

PYTHONPROFILEIMPORTTIME=1 python3 myscript.py 2>1.log

tuna

然后 tuna 1.log ,会自动打开浏览器围观 import 耗时分布。

Posted

stdout

FastAPI/Starlette支持静态文件支持SPA

FastAPI 官方支持 from fastapi.staticfiles import StaticFiles 充当一个静态文件服务器

其实实现是 starlette。这玩意可以在 directory 下放一个 404.html,恰好单页应用也需要用 index.html 充当所有 javascript 框架注册的 router

只是有一个毛病,这货返回的 HTTP status code 是 404。用起来没啥大毛病,但是就是浏览器不会记录网址,导致没法匹配浏览历史快速找到之前访问过的页面。

拿来改改继续用

  from starlette.staticfiles import StaticFiles, Scope, Headers, Response, FileResponse

  class StaticFilesWithout404(StaticFiles):
      async def get_response(self, path: str, scope: Scope) -> Response:
          r = await super().get_response(path, scope)
          h = Headers(scope=scope)
          full_path, stat_result = await self.lookup_path('404.html')
          if isinstance(r, FileResponse) and r.path == full_path:
              if 'text/html' in (h.get('accept') or ''):
                  r.status_code = 200
              else:
                  return Response('', status_code=404)
          return r

  app.mount("/frontend", StaticFilesWithout404(directory="frontend", html=True), name="static")

这段代码大概是起到了类似 nginx try_files 的作用。默认静态文件映射一个目录,但是如果找不到就按根目录的 index.html 输出。这里还判断了请求的 accept 头,如果是 xhr/Fetch 就会直接返回一个0字节长度无MIME的404。

由此我想到了一个学究式的 leaky abstraction。众所周知,如果一个URL不存在,那么服务器应该返回404

但是现在都是 SPA 单页应用,都是先 200 返回 index.html 的内容,再由 javascript 的 router 去决定是否正常显示还是404 。所以返回 200 但是显示 404 就破坏了这个语义。如果要精确匹配SPA里的router如果不存在由服务器返回404,第一是搞 SSR,把前端那一坨代码跑在服务器端,第二种办法是 npm build 的时候输出一个可路由URL列表,然后部署的时候导入静态文件服务器更新。这个具体实现就留作homework由读者自行解决了。

Posted

stdout

人工构造Flask session模拟cookie登陆

有没有好奇为什么 Flask 配置必须要求一个 SECRET_KEY,然后就可以在浏览器保存一个 session 状态读写数据。

这里记一下它的底层实现,其实需要依赖的包是 itsdangerous

  import hashlib
  from itsdangerous import URLSafeTimedSerializer
  URLSafeTimedSerializer(
          'YOUR_SECRET_KEY',  # flask SECRET_KEY
          'cookie-session',  # from flask.sessions.SecureCookieSessionInterface.salt
          # serializer=TaggedJSONSerializer(),
          signer_kwargs={'key_derivation': 'hmac', 'digest_method': hashlib.sha1}
  ).dumps({
      "your_key": "your_value"
  })

别人如果拿到你的 SECRET_KEY 就可以伪造任意 session cookie 了

Posted

stdout

Python 3.X ctypes 和 greenlet size changed 坑三则

安装 setup.py 的时候 No module named '_ctypes' 报错

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/local/python3/lib/python3.9/site-packages/setuptools/__init__.py", line 18, in <module>
      from setuptools.dist import Distribution
    File "/usr/local/python3/lib/python3.9/site-packages/setuptools/dist.py", line 34, in <module>
      from setuptools import windows_support
    File "/usr/local/python3/lib/python3.9/site-packages/setuptools/windows_support.py", line 2, in <module>
      import ctypes
    File "/usr/local/python3/lib/python3.9/ctypes/__init__.py", line 8, in <module>
      from _ctypes import Union, Structure, Array
  ModuleNotFoundError: No module named '_ctypes'

解决办法是不要用 3.9。

如果实在要用,则自己编译前需要加上 yum install libffi-devel 或者 sudo apt-get install libffi-dev

来自SO

greenlet.greenlet size changed 然后出错

/usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)
/usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
  return f(*args, **kwds)

Segmentation fault (core dumped)

这个摘自官方 iseue

On Python 3.7 and higher, gevent 20.9.0 is required to use (a standard build of) greenlet 0.4.17.

解决办法:pip install gevent==20.9.0 pip install greenlet==0.4.17

SQLAlchemy 1.4 大战 greenlet 0.4.17

sqlalchemy changelog

to accommodate for the handling of Python contextvars (introduced in Python 3.7) for greenlet versions greater than 0.4.17. Greenlet version 0.4.17 added automatic handling of contextvars in a backwards-incompatible way; we’ve coordinated with the greenlet authors to add a preferred API for this in versions subsequent to 0.4.17 which is now supported by SQLAlchemy’s greenlet integration. For greenlet versions prior to 0.4.17 no behavioral change is needed, version 0.4.17 itself is blocked from the dependencies.

然后就直接在 setup.cfg里greenlet != 0.4.17了。

神仙打架,不听不听,和尚念经,果断 sqlalchemy==1.3.24 降级。

Posted

stdout

Edge调教指南——如何设置本地PAC代理和首页天气

之前一直用的chrome v70,为什么呢?

第一是这个老版本允许 --proxy-pac-url="file:///Users/me/1.pac 这样设置,但是Chromium项目的大爷们觉得你本地的.pac不够安全,要网上的.pac才安全,所以一刀切给禁了。当然有热心人士觉得这没啥大不了的。不外乎装个插件就可以切PAC。但是chrome插件其实会有一个fingerprint。Extension一启用,隔壁老王都知道你开代理了。

第二,也是最重要的原因,这是最后一个chrome版本支持 about://net-internals 在本地查看浏览器底层网络请求。现在无论是 Edge 还是 Chrome 都必须把 .json 文件从 https://netlog-viewer.appspot.com/ 加载才能看了。很尼玛保护了隐私有没有啊。

但是坚持老版本,也付出了巨大的代价。很多新的 ES 语法不支持,比如说 Grafana 最新版 login 都进不去。这就尴尬了。只能被自愿升级了。

既然升级是必选项,那么就换 Edge 试试。

第一就是得解决这个本地 .pac 问题,其实从上面第二个工具里才能知道,PAC加载失败有个返回 ERR_DISALLOWED_URL_SCHEME。stackoverflow 上找到个奇技淫巧

open "/Applications/Microsoft Edge.app" --args --proxy-pac-url='data:application/x-javascript-config;base64,'$(base64 -i /Users/me/2.pac)

还能这样玩?双击666。

第二个把该禁用的禁用了,首页只留一个搜索框一个天气。但是这天气就特么定位到坡县了。而且下拉框点不出来啊。F12打开 devtool 一看,我尼玛好家伙 https://www.bing.com/api/v6/Places/AutoSuggest 被 Bing China 的网管给一把梭302到 https://cn.bing.com/api/v6/Places/AutoSuggest 了。然后就特么 CORS 给拦截,js读取不到列表。。。

最后怎么办?点那个天气图标,进入到 https://www.msn.com/en-sg/weather/today/weather-today/we-city

然后找那个地图,手动选择自己所在城市,设置为 Home 。等 cookie 写入生效了,打开edge新tab就ok了

这年头上个网真鸡儿累。

btw 设置M$账号同步的时候,发现账号所在国家可以选很多不是国家的地方,但是时区就不能随便选了。F12改了也没用。微软的后台开发做的过滤还挺细致的。

btw2 发现一个很干净的启动页: chrome-search://local-ntp/local-ntp.html 或者直接在 hosts 里把 ntp.msn.com 干掉,有奇效。

Posted

stdout

TiDB不支持JSON字段默认值

继前面的日期不兼容坑,又发现一个

The BLOB, TEXT, and JSON columns cannot be assigned a default value.

虽然 MySQL 5.7 也是这么说的:

A JSON column cannot have a non-NULL default value.

但是 MySQL 8.0.13 就支持指定默认值

Prior to MySQL 8.0.13, a JSON column cannot have a non-NULL default value.
The default value specified in a DEFAULT clause can be a literal constant or an expression. With one exception, enclose expression default values within parentheses to distinguish them from literal constant default values. Examples:

虽然语法很丑但是很实用啊。

所以这种语句

  ALTER TABLE mytable MODIFY COLUMN my_json JSON DEFAULT (json_object());

在 TiDB 里就没法支持。这也带来一个问题,就是要 upsert 这个 字段的时候,写起来就很拧巴,以 peewee 这个ORM为例:

  MyTable.update({
      'my_json' = fn.json_set(
          fn.COALESCE(MyTable.my_json, Cast('{}', 'JSON')),
          '%.some_key', 'some_value',
  )})

必须用 COALESCE 预防默认值为 NULL 的情况。太麻烦了。

btw 玩MySQL/TiDB的 JSON 有个坑的,那就是 json 的 null 是有值的,和 sql 的 NULL 是不同的。

也就是说

  select json_extract('{}', '$.hey') IS NULL;
  select json_extract('{"hey": null}', '$.hey') = Cast('null' as JSON);

要注意。

btw2 MySQL的JSON数据抽取 ->> 是返回真正的值,等价于TiDB里的 json_unquote(json_extract(t.json_field, '$.key1.key2'))

Posted

stdout

Chrome 指定域名解析,绕过 hosts

看到 V站有人问这个,随手一记

可以参考源码里的注释文档

--host-resolver-rules=rules

Like --host-rules but these rules only apply to the host resolver.
A comma-separated list of rules that control how hostnames are mapped.
For example:
- MAP * 127.0.0.1 Forces all hostnames to be mapped to 127.0.0.1
- MAP *.google.com proxy Forces all google.com subdomains to be resolved to proxy.
- MAP test.com [::1]:77 Forces test.com to resolve to IPv6 loopback. Will also force the port of the resulting socket address to be 77.
- MAP * baz, EXCLUDE www.google.com Remaps everything to baz, except for www.google.com.

These mappings apply to the endpoint host in a net request (the TCP connect and host resolver in a direct connection, and the CONNECT in an HTTP proxy connection, and the endpoint host in a SOCKS proxy connection).

官方文档有语焉不详的提到过

https://chromium.googlesource.com/chromium/src/+/refs/heads/main/net/dns/README.md

Starting Chrome with --host-resolver-rules="MAP the.hostname.com [dead::beef]" where the.hostname.com is the hostname to allow resolving and dead::beef is the IPv6 address to resolve it to. net::MappedHostResolver acts at a level before IPv6 connectivity checks, and if a hostname is remapped to an IP literal, connectivity checks do not apply.

还有很多 proxy的参数

curl 的参数

也顺便贴一下。调试的时候用得上

参数:

  • -4, --ipv4
  • -6, --ipv6
  • --dns-interface <interface>
  • --dns-ipv4-addr <ip-address> Tell curl to bind to <ip-address> when making IPv4 DNS requests
  • --dns-ipv6-addr <ip-address> Tell curl to bind to <ip-address> when making IPv6 DNS requests
  • --dns-servers <ip-address,ip-address> Set the list of DNS servers to be used instead of the system default. The list of IP addresses should be separated with commas. Port numbers may also optionally be given as :<port-number> after each IP address.
  • --resolve <host:port:address> Make the curl requests(s) use a specified address and prevent the otherwise normally resolved address to be used. Consider it a sort of /etc/hosts alternative provided on the command line. The port number should be the number used for the specific protocol the host will be used for. It means you need several entries if you want to provide address for the same host but different ports. This option can be used many times to add many host names to resolve. (Added in 7.21.3)

最后一个很有用 😹

Posted

stdout

reverse proxy FastCGI (fcgi) with uWSGI

People are still using FastCGI, and it's hard. I used uWSGI to convert FCGI traffic to regular http/1.0 so I can keep my existing http service running without apache mod_fcgi shit.

CGI environ variables

This is some ancient knowledge, better keep them noted

KEY description
AUTH_TYPE ignore this
REMOTE_IDENT The user making the request.
REMOTE_USER The authenticated name of the user making the query.
CONTENT_LENGTH byte length of body
CONTENT_TYPE MIME such as text/html, without the charset shit
DOCUMENT_ROOT NEW. The directory from which web documents are served.
REQUEST_URI NEW. confusing shit similar with PATH_INFO
HTTP_* HTTP headers
PATH_INFO like /a/b/c
PATH_TRANSLATED not really used
QUERY_STRING the part after ? in an URL
REMOTE_ADDR client IP
REMOTE_HOST client host name by lookup ip
REQUEST_METHOD GET/POST
SCRIPT_NAME The virtual path (e.g., /cgi-bin/program.pl) of the script being executed.
SCRIPT_FILENAME confusing shit similar with SCRIPT_NAME
SERVER_NAME The server's hostname or IP address.
SERVER_PORT The port number of the host on which the server is running.
SERVER_PROTOCOL The name and revision number of the server protocol.

reverse proxy FastCGI with uWSGI

slap this into my_conf.ini

  [uwsgi]
  my_port = 9000  # can be set otherwise
  route-run = addvar:SERVER_NAME=%h
  route-run = addvar:SERVER_PORT=%(my_port)
  route-if  = empty:${REQUEST_METHOD} addvar:REQUEST_METHOD=GET
  route-if  = empty:${HTTP_HOST} addvar:HTTP_HOST=%h:%(my_port)
  route-run = seturi:/fcgi-proxy${PATH_INFO}
  route-run = http:127.0.0.1:9527
  fastcgi-socket = 0.0.0.0:%(my_port)
  http-socket = 0.0.0.0:8000  # for testing purpose.

Install uWSGI, install the binary from pypi wheels instead of compiling uwsgi.c shit

  pip install pyuwsgi

Run it

  uwsgi --ini my_conf.ini

Now you can run your regular http service behind 127.0.0.1:9527, while serving fcgi on 127.0.0.1:9000

Requests like fcgi://127.0.0.1:9000/hello will be fowwarded to http://127.0.0.1:9527/fcgi-proxy/hello

You may ask why adding a /fcgi-proxy in path? Because there's a gotcha.

fuckedup FastCGI requests

Learned this from some php-fpm tutorial, first install the standard fcgi client tool because curl doesn't speak fcgi.

  apt-get install libfcgi0ldbl
  yum --enablerepo=epel install fcgi
  SCRIPT_NAME=/ping SCRIPT_FILENAME=/ping REQUEST_METHOD=GET cgi-fcgi -bind -connect 127.0.0.1:9000

You can observe from uWSGI logs that cgi-fcgi command sends flawed requests, the PATH_INFO was unset.

After hours wasted, I came up with a clever hack to force uWSGI's internal router to force seturi with a path prefix

ToDo

There are further problems that can be improved, for example, my http service is running uvicorn, it can not guess the client ip:port infor from its stupid get_remote_addr

what's worse, the uWSGI addheader directive are for responses, so I can't add request headers like x-forwared-for.

Also the reverse proxy is HTTP/1.0 only. to make the Connection: Keep-alive happen needs some further investigation.

Q&A

Thanks for reading my shitpost. Questions?

Posted

stdout

淘宝同一手机最多绑定6个小号

总结一些没用的知识

  1. 淘宝/阿里账号,可以一个手机最多绑定6个账号。淘宝注册的时候点击「企业注册」然后用邮箱注册,最后用同一个手机验证即可

  2. 「支付宝」在淘宝/阿里系里app里是和「账号」一一绑定的,这就产生了很多冲突。比如你一个第三方外卖app,支付的时候可以选择 微信/支付宝,一般不会限制下单手机号和支付手机号是否一样,你甚至可以帮别人支付订单,但是淘宝/阿里的app不一样,只能自己账号支付自己账号的订单。如果你有两个手机号和两个支付宝账号,那么没法交叉支付。

还有一个想不起了,想起了再补充。2333

Posted

stdout

《兵车行》竖排实验

群里闲聊起究竟生男好还是生女好,就想起了这首诗

兵车行——杜甫 车辚辚,马萧萧,行人弓箭各在腰。 耶娘妻子走相送,尘埃不见咸阳桥。 牵衣顿足拦道哭,哭声直上干云霄。 道旁过者问行人,行人但云点行频。 或从十五北防河,便至四十西营田。 去时里正与裹头,归来头白还戍边。 边庭流血成海水,武皇开边意未已。 君不闻,汉家山东二百州,千村万落生荆杞。 纵有健妇把锄犁,禾生陇亩无东西。 况复秦兵耐苦战,被驱不异犬与鸡。 长者虽有问,役夫敢申恨? 且如今年冬,未休关西卒。 县官急索租,租税从何出? 信知生男恶,反是生女好。 生女犹得嫁比邻,生男埋没随百草。 君不见,青海头,古来白骨无人收。 新鬼烦冤旧鬼哭,天阴雨湿声啾啾!

测试下 css 的文字竖排。

字体来自 fonts.css

Posted

stdout

Re-index page on Google Search Console

Request indexing webpage with Google used to be fun and simple, now it's a tedious task especially when you have tons of old pages ready to be crawled yet Google is reluctant to move.

Here is a simple script to speed up the stacks of page you have click in order to submit multiple missing pages to Google.

Open your Google Search Console, click Coverage then open Discovered – currently not indexed

Now open your browser devtool, paste these into javascript console, hit them one by one.

  document.querySelector('div[data-disabled] table td span[title] span[title="Inspect URL"]').click()

  document.querySelector('span[data-event-action=request-indexing] span[jsslot] div span span').click()

  document.querySelector('span[data-event-action=pivot-to-report-main-view-indexing-section] a').click()

Until you met with the daily quota. Rinse and try this shit everyday.

Fuck Google.

Posted

stdout

Letsencrypt 在老版本macOS Ubuntu 上修复TLS证书错误

一觉醒来,打开浏览器,各种连不上,调试了一下发现说lets encrypt 的 acme https证书错误

仿佛记得V站讨论过这个事,以为和自己无关,结果也中招了。DST Root CA X3 根证书在国庆前过期了。

老版本 Ubuntu 修复:

  sudo apt update
  sudo apt install ca-certificates

老版本 macOS 修复:

需要手动导入并信任新的 ISRG Root X1 根证书。来自https://letsencrypt.org/certificates/

保存下列为 1.pem

  -----BEGIN CERTIFICATE-----
  MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
  MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
  DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
  TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
  cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
  AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
  ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
  wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
  LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
  4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
  bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
  sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
  Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
  FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
  SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
  PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
  TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
  SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
  c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
  +tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
  ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
  b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
  U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
  MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
  5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
  9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
  WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
  he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
  Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
  -----END CERTIFICATE-----

打开 keychain access,把这个 1.pem 拖入 system,双击打开,在 Trust 这个下拉,选择 信任。

因为浏览器没法用,所以搜索资料也麻烦。真是差点锁死。

Posted

stdout

美国自来水 TDS 这么高?

看到个讲 Solar Roof 视频

他们家的 filtered tap water 测出来 TDS 为 358ppm

tap water 测出来 TDS 是 435 ppm,

说加州的市政供水很硬,有大量的钙、镁离子。都不敢直接拿来洗车,怕喷枪给表漆洗得坑坑洼洼;他们家太阳能光伏板很脏想冲也不敢,万一刮坏了不值。加州又是天天大太阳不下雨,还好他们家有两个大水桶接雨水,测出来TDS 20,刚好拿来冲屋顶。

这个说法令我很吃惊的。因为一直以来网上和一些老外都说中国自来水差什么的。美国水龙头打开直接喝。翻了一下,留学生论坛都说美国的自来水很硬,会掉头发什么的,看来 TDS 真的很高啊。

为啥网上会有「美国自来水水质很好水龙头打开直接喝」这个说法?翻了一些帖子,感觉应该是这样,老中过去留学、旅游的都是住旅店、公寓,其实一般楼栋那都统一装了净水装置。公园、学校、图书馆的直饮机其实也装了过滤装置的。所以看上去能「直接喝」而已。如果你自己装修楼,接市政供水,那么大家都是要装初滤,加 Pentair 大蓝瓶的。

国内自来水,我家 TDS 测出来是130-150左右。按照 WHO饮用水 的标准,属于口感极佳。

注:水的 1 mg/L 约等于 1 parts per million (ppm)

我也装了一套 RO 过滤的,TDS 测出来 5。娃哈哈桶装水测出来是 0。厉害!

网上还有个说法「美国直饮水标准为≤ 50mg/L,欧盟直饮水标准为≤ 70mg/L」,注意说的是「直饮水」,不是自来水。查了下,美国 EPA 对饮用水 TDS 其实没有强制要求,EPA有个二级饮用水非强制标准CFR143,推荐 TDS 小于 500 mg/L。这个推荐标准是不是定得太高了?作为对比,国标《生活饮用水卫生标准GB5749-2006》是 1000。拉垮!

最后说下,水质的评价有很多指标,我这里是不是唯 TDS 论了?是,也不是。如果都是过滤后的水,那么 TDS 这个指标是非常有代表性的。水里的杂质体积从大到小是 悬乎颗粒物 > 微生物 > 细菌病毒 > 离子。TDS测导电率,是离子的数量。如果体积最小的离子在过滤后都下降了,那么别的杂质其实也没剩多少了。所以这个指标对于「过滤水」来说是很有意义的。

pps 看到个轶事

在上世纪初,人们还不知道水污染与流行病的关系,也没有可靠的自来水厂,加州曾经爆发过大规模的流行病,当时人们发现很多华人聚集区发病率显著低于其他人口,最初以为是华人有某些特殊的抗病能力(就如同印第安人缺少抗感冒病毒的能力,本文作者引申),后来才发现是因为劳工中普遍保持着烧开水喝的习惯,避免了流行病在华人间的传播。然后才开始了大规模建设净水厂,通过去除悬浮颗粒及胶体,再进行消毒,有效灭活细菌,从而控制了水介质病原菌的传播,带来了水处理行业的发展(本段内容依据自加州大学伯克利分校Nelson教授的课堂讲述,本文作者根据其大意进行概括)。

Posted

stdout

TiDB 的日期比较和 MySQL 不兼容

MySQL的

  > select VERSION()
  +-----------+
  | VERSION() |
  +-----------+
  | 8.0.15-6  |
  +-----------+


  > select NOW() > '';
  +------------+
  | NOW() > '' |
  +------------+
  | 1          |
  +------------+

  > show warnings;
  +---------+------+----------------------------------------+
  | Level   | Code | Message                                |
  +---------+------+----------------------------------------+
  | Warning | 1292 | Truncated incorrect datetime value: '' |
  +---------+------+----------------------------------------+

TiDB:

  > select VERSION()
  +--------------------+
  | VERSION()          |
  +--------------------+
  | 5.7.25-TiDB-v5.1.0 |
  +--------------------+
  1 row in set

  > select NOW() > '';
  +------------+
  | NOW() > '' |
  +------------+
  | <null>     |
  +------------+

  > show warnings;
  +---------+------+------------------------------+
  | Level   | Code | Message                      |
  +---------+------+------------------------------+
  | Warning | 1292 | Incorrect datetime value: '' |
  +---------+------+------------------------------+
  1 row in set

记录一下 相关 issue

btw 顺便吐槽一下,

json_extract('{"a": null}', '$.a') 返回的值是 CAST('null' AS JSON)

注意区分 json_extract('{"a": null}', '$.b') IS NULL

Posted

stdout

shell script to graceful restart gunicorn with source code reloading

I always thought restart gnicorn is as easy as kill -HUP <pid>, but there are two pitfalls:

First of all the code might be loaded in master process then fork'ed into workers, so the HUP signal won't reload the source code, only gunicorn config is updated.

Then there might be some bugs and your new code won't boot correctly, the worker will quit and drags down the master process along with it.

I discovered a new trick from github issues that can spawn new worker first, check if new code loads correctly then restart the whole rack.

According to the docs you need to send several signals and blah blah, the author tried to explain in great detail of how it works rather than providing a simple option to do so.

So I wrote a shell script to do that instead.

  #!/bin/bash

  oldpid=$(<./deploy/gunicorn.pid)
  # forget the old kill -HUP 
  kill -USR2 $oldpid
  sleep 1
  timeout=5
  while [[ $timeout -gt 0 ]]; do
    newpid=$(<./deploy/gunicorn.pid.2) && kill -0 $newpid
    if [[ $? -eq 0 ]]; then
      break
    else
      sleep 0.5
      (( timeout-- ))
    fi
  done;

  # shutdown old
  echo "success. newpid=$newpid"
  kill -WINCH $oldpid
  sleep 1
  kill -TERM $oldpid

The script will check for new master pid to be alive then kill the old master in a maximium timeout of 5 seconds.

It works perfectly to replace the simple HUP signal as intended.

Posted

stdout

macOS 设置 TCP 握手连接超时,提升浏览器在糟糕网络下的访问体验

多年的疑惑今天居然被偶然解开了。

由于墙内糟糕的网络环境,而且ISP甚至wifi内还有墙中墙,所以很多域名是打不开的。

每次看浏览器在傻乎乎的转菊花,就觉得浪费时间。

  sudo sysctl  net.inet.tcp.keepinit=3000

这一条指令可以直接设置 macOS 的 TCP Keepalive 心跳包初始化时间设置为3000毫秒。

net.inet.tcp.keepinit默认值是 75000 ,单位是毫秒。注意 Linux 下没这设置,而且单位是 秒

这句话学究式的解释可以通过 man 4 tcp 得到,摘录几条:

TCP_CONNECTIONTIMEOUT The TCP_CONNECTIONTIMEOUT option allows to specify the timeout, in seconds, for new, non established TCP connections. This option can be useful for both active and passive TCP connections. The default value is specified by the MIB variable net.inet.tcp.keepinit.
tcp.keepinit Timeout, in milliseconds, for new, non-established TCP connections.

换句话说,就是 TCP 第一个 syn 包最多给3秒等待。超过3秒就拉倒,直接报 ERR_CONNECTION_TIMED_OUT 完事。

我觉得,如果一个域名和TCP端口3秒钟内无法完成握手,就别执着了。你再干等75秒也不可能连上。这个值只会影响初始连接的超时,一旦连上了怎么idle都不影响,所以不会影响游戏的tcp或者websocket的长连接。也不影响udp。

2021-07-06 补充:

重启保持有效:

  echo net.inet.tcp.keepinit=3000 | sudo tee -a /etc/sysctl.conf

2021-09-21 补充:

新版macOS 把 /etc/sysctl.conf 这文件给换位置了。应该是

  "/Users/Shared/Relocated Items/Configuration/private/etc/sysctl.conf"

Posted

stdout

MySQL on duplicate key in SQLAlchemy

A simple function to upsert something atomically into MySQL

  def upsert(val1, val2, **kwargs):
      """upsert val1-val2 pair into mysql and return the pk"""
      from sqlalchemy.dialects.mysql import insert
      from sqlalchemy.sql.expression import func

      # this val1-val2 pair should have UNIQUE KEY constraint in MySQL table.
      stmt = insert(MyModel.__table__).values(col1=val1, col2=val2)
      if kwargs:
          stmt = stmt.on_duplicate_key_update(kwargs)
      else:
          # get lastrowid https://stackoverflow.com/a/29722203/41948
          stmt = stmt.on_duplicate_key_update(id=func.LAST_INSERT_ID(MyModel.id))
      # inserted_primary_key (dbapi, return 0 when only insert) or lastrowid (mysql)
      pk = db.session().execute(stmt).lastrowid
      if not pk:
          pk = MyModel.query.options(load_only('id')).filter_by(
              col1=val1, col2=val2
          ).first().id
      return pk

Writing complex SQL is hard, sqlalchemy makes it harder. Prefer other ORMs in Python.

I really hate fighting the infamous sqlalchemy API. Everything wrong about it can be shown in this one-liner:

  from sqlalchemy.orm import Load, load_only, joinedload

You have the three styles of CamelCase, under_score, and concatfunctionnames fucked up together. Bad and leaky abstraction.

Posted

stdout

古墓丽影9(Tomb Raider 2013)治3D眩晕必备FOV补丁+禁镜头晃动

断断续续把古墓丽影9打完了。这款2013年的游戏放在今天看也很不错。

唯一的巨大问题就是3D头晕。网上找了一些办法。

禁止镜头晃动

虽然故事和镜头都很好,但是这晃动太烦人了,很容易头晕。值得庆幸的是 PC 平台可以彻底根治这问题。下载 Yamatai Patch。按图修改

增大 FOV

这是多年的经验。头晕的主要问题就是视野太窄。

去下载 Cheat Engine 安装并打开,然后下载内存修改表,按图修改。

以上修改都需要有管理员权限并且保证游戏正在进行中。顺便说下 Yamatai patch 还可以治好 Lara Croft 胸部发育不良的毛病。好顶赞!

Posted

stdout

行车记录仪视频ffmpeg合并转录存档,跳过重复的漏秒

手上有几个行车记录仪

  • VOSONIC 勤宇V10,也叫群华 路不平,用的 联咏 Novatek NT96550BG 的芯片,1080p
  • BLACKVIEW 凌度A12,采用 安霸(Ambarella) A12 方案,2k
  • PAPAGO 趴趴狗,GoSafe 560WiFi,采用 NT96670 芯片+IMX415的 sony CMOS,4k

特点是每5分钟切割成一个单独的文件,为了防止每两个视频之间漏秒,所以会额外重复写入1秒的内容

现在想把一段时间的视频合并成一个存档,并且要跳过那些额外的1秒,上 ffmpeg。

首先前摇,生成需要合并的视频文件路径列表

ls -fd1 /Volumes/SD_CARD/DCIM/xxxxx/*

的结果写成如下格式:

  file 'input1.mov'
  inpoint 10
  outpoint 300
  file 'input2.mov'
  # comment
  outpoint 300 
  file 'input3.mov'
  outpoint 300
  ... 
  file 'inputN.mov'
  outpoint 300

inpoint 10 就是从10秒开始算,outpoint 300 就是到300秒结束。保存上面的为 1.txt

然后念 ffmpeg 咒语

ffmpeg -hwaccel cuda -hwaccel_output_format cuda -safe 0 -r 48 -f concat -i 1.txt -an -vf "select=concatdec_select,scale_cuda=w=1280:h=720:interp_algo=lanczos,setpts=0.125*PTS" -c:v h264_nvenc -preset:v p7 -profile:v high -tune:v hq -cq:v 19 -qmax 22 -b:v 0 -maxrate 8M -rc vbr out.mp4

逐个解释下:

  • -hwaccel cuda 用nVIDIA的 H.264 解码器硬件加速解码行车记录仪的原始 .MOV 文件。如果没显卡有 Intel 处理器可以 把 cuda 换成 qsv 也有加速效果。也可以换成 cuvid 但是速度更慢
  • -hwaccel_output_format cuda 生成 out.mp4 的时候也采用 nVIDIA 显卡H.264编码器加速。这里如果和上面都是 cuda 那么直接显存对拷,避免了内存和 PCI 拷贝速率瓶颈。
  • -safe 0 允许读写外部文件
  • -r 48 和后面的 setpts=0.125*PTS 一起,把帧率提高到 48fps ,把回放速度提高 8倍。方便存档。不需要的可以去掉
  • -f concatselect=concatdec_select 就是通过前摇的 1.txt 指定合并哪些文件(并且去掉时间起止点之外的)
  • -an 丢掉音轨。如果保留的话应该是 -v:a copy
  • -vf 后面指定一系列巨复杂的ffmpeg滤镜操作。
  • scale_cuda=w=1280:h=720:interp_algo=lanczos,用显卡 CUDA 操作缩放操作。lanczos是一种 downsampling 算法让画面更加平滑。如果你自己编译的还可以把 scale_cuda 换成 scale_npp 进一步提速。
  • 如果没有独显换成软缩放 -vf "select=concatdec_select,scale=1280x720,setsar=1,setpts=0.125*PTS" -sws_flags lanczos
  • -c:v h264_nvenc 视频压缩引擎:nVIDIA 的 H.264 编码器。如果没有独显可以改成 libx264
  • -preset:v p7 这里代替之前的 slow fast 等选项。参考 HEVC Preset Migration
  • -profile:v high 如果你要兼容初代 iPad 等古董设备就只能用 main
  • -tune:v hq 面向画质优化(而不是码率或者延时)
  • -cq:v 19 画质非常接近无损的。但是体积也变大。一般22中等配置不能再大了。H.265 可以设置成 26。如果是 libx264 改成 -crf 19 指定
  • -qmax 22 上面的值最大为 22
  • -b:v 0 取消码率指定
  • -maxrate 8M 最高码率 8Mbps
  • -rc vbr 可变码率。注意 vbr_hq 之类的已经不在支持了。

还有一些坑点:

  • 2k@60fps的分辨率需要指定 -level 51。因为默认的 H.264 级别太低frame size支持不了。。。
  • 如果不缩放原始尺寸压制,很有可能会出错 No decoder surfaces left. Error while decoding stream #0:0: Invalid data found when processing input。这是 ffmpeg 4.1 以后的 bug 解决方案是 -extra_hw_frames 2 或者更大。但是 CUDA 最多支持32所以估计会报错,就只能 CPU 负责解码 GPU 负责编码。
  • 压制的时候发现 TF 卡或者是读卡器20M/s是瓶颈。拷到本地发现 GPU decoder 是100%,encoder只有30%左右。
  • 2k分辨率20Mbps码率30fps的原始视频占用体积太大,1080p 16Mbps码率效果比 720p 8Mbps好不了多少。最后选择的是8M码率。在超快移动的时候有色块,其他堵车的时候画质很高。识别车牌几乎不受影响

总的来说,ffmpeg 真是操作糟糕而功能强大的工具的典型。以上参数为东拼西凑撞大运偶得,如有不对敬请指正。

Posted

stdout

Win10 自动设置开机锁屏壁纸

Win10启动后,锁屏界面有一些每天更新的漂亮的墙纸,但是每个用户进去却只能看到默认的墙纸,不能自动切换。很是遗憾,于是搞了一段小脚本弥补。

找个地方比如桌面,右键新建一个 est_win10_auto_wallpaper.bat

  @if (@X)==(@Y) @end /* JScript comment
  @echo off

  cscript //E:JScript %~n0.bat

  exit /b %errorlevel%
  */

  var oShell = new ActiveXObject("WScript.Shell") ;
  var oFSO = new ActiveXObject("Scripting.FileSystemObject");

  var sWinDir = oFSO.GetSpecialFolder(0);
  var sPath = oShell.ExpandEnvironmentStrings('%LOCALAPPDATA%\\Packages\\Microsoft.Windows.ContentDeliveryManager_cw5n1h2txyewy\\LocalState\\Assets\\')

  var oFolderEnum = new Enumerator(oFSO.GetFolder(sPath).Files);
  var aFiles = []
  for (;!oFolderEnum.atEnd();oFolderEnum.moveNext()) {
      // var item = oFolderEnum.item()
      aFiles.push(oFolderEnum.item())
  }
  var aFiles2 = aFiles.sort(function(x,y){return new Date(y.DateLastAccessed).getTime() - new Date(x.DateLastAccessed).getTime()}).slice(3)
  var sWallpaper;
  // for (var i=0;i<aFiles2.length;i++){
  for (var i in aFiles2){
      // WScript.Echo('check ' + aFiles2[i].name + ' size: ' + aFiles2[i].size);
      if(aFiles2[i].size > 500000){  // guess > 500KB is good wallpaper
          sWallpaper = aFiles2[i].path
          break;
      }    
  }
  if(sWallpaper){
      // or fso.GetStandardStream(1) for stdout, 2 for stderr
      // WScript.Echo('set to ' + sWallpaper);
      oShell.RegWrite("HKCU\\Control Panel\\Desktop\\Wallpaper", sWallpaper);
      oShell.Run("%windir%\\System32\\RUNDLL32.EXE user32.dll,UpdatePerUserSystemParameters 1, True");
  }

Win+R输入 shell:startup 回车,把刚才的 est_win10_auto_wallpaper.bat 在这里创建一个快捷方式

解释下为啥要在 shell:startup 里创建快捷方式而不是直接放 .bat 。因为快捷方式可以设置当前运行目录,这样代码里的 %~n0 才能生效。懒得查询完整路径如何写了。。

然后这段代码其实用了一个比较高级的 .js .bat 混合格式。因为 Windows Smart Screen 会拦截 WScript 的直接执行,认为特别危险。但是如果你在 cmd 里执行系统就觉得很安全。。。。嗯。。

吐槽1: 开始 → 启动 菜单都不见了。。。

吐槽2:好久没写 WScript 了感觉都忘完了。vbs 是肯定不会的了。JScript 居然是 ESMAScript3。好多现代写法都不支持。

Posted

stdout

Mac 命令行终端里获得视频文件的长度

记录一下,Mac OS X 10.6.8 或更高版本:

  mdls -name kMDItemDurationSeconds -name kMDItemFSName  1.mp4

这命令有一个很神经病的一点,同一个 .mp4 文件,获取只读的 TF 卡上的就没法获得视频长度,复制到可以读写的本地硬盘上就可以读出来。

或者在 Finder 里新建一个叫 Movies 的文件夹,把视频文件放进去,然后列表展示,表头右键,就可以选择显示长度。。

同理,建立一个叫 Pictures 的文件夹,可以列表展示图片尺寸。。

来自 superuser 。真是神一般的设定。

Posted

stdout

FastAPI/Starlette+Requests实现反向代理

因为某些拉垮的业务需要,不得不在代码里去反向代理别的 HTTP API

一般格式如下:

  @app.get('/other/{other_path:path}')
  @app.post('/other/{other_path:path}')
  async def other_api(other_path: str, req: Request):
      """透传 API"""
      host = 'http://example.intranet'
      url = '{}/other/{}'.format(host, other_path)
      body = bytes(await req.body()) or None
      r = requests.request(
          req.method, url,
          headers={
              'Cookie': req.headers.get('cookie') or '',
              'Content-Type': req.headers.get('Content-Type')},
          params=req.query_params, data=body, stream=True,
          allow_redirects=False)

      h = dict(r.headers)
      h.pop('Content-Length', None)
      return StreamingResponse(r.raw, headers=h, status_code=r.status_code)

解释一下为啥要这么写:

  1. 采用 stream=True + StreamingResponse,防止上游给你返回一个GB体量的文件把 FastAPI 进程内存撑爆
  2. 只摘取浏览器的 Cookie + Content-Type 两个头。别的不给上游。
  3. 要去掉返回的 Content-Length 头。因为 StreamingResponse 会自己定义返回长度。

这样写基本能跑起来了,但是有2个问题:

  1. 如果上游返回一个 302/303/307 的跳转,那么很有可能把内网的跳转网址直接返回给浏览器了。因为 HTTP/1.1 的标准 RFC2616 强制规定,Location 的值必须是一个绝对网址。所以这个时候就冒犯一下这个规定,强行截断:
        loc = h.pop('Location', '')
        if loc.startswith(host):
            h['Location'] = loc[len(host):]
    

注意这里不要用 .lstrip() 。这方法和你想象的完全不是一回事。它是基于单个字符的挨个替换而不是整个字符串。
2. 实测这种在业务代码实现的「软」反向代理性能堪忧。上游4s左右的整个页面加载时间,反向代理之后就变成了 120s 左右。盲猜是 StreamingResponse 是一个字节一个字节去遍历迭代器?实测并不是。主要的锅还是 Requests.raw 是按照 chunked 一段一段返回的。最好还是准备个 4KB 的 buffer,提高性能。解决方案是把 r.raw 改成 r.raw.stream(4096000) 性能一下就高了。注意这里不要用官方文档提供的 .iter_contents() 或者 .iter_lines() 这两者又会自作聪明的去解析文本编码或者换行符,降低了性能。

Posted

stdout

OpenWRT 实现 Cloudflare 动态域名 dynamic DNS

首先得有公网 IP (废话),然后你得 OpenWRT 路由器桥接直拨。

这个方法的核心是通过 ifstatus wan | jsonfilter -e '@["ipv4-address"][0].address' 命令得到本机 WAN 的公网 IP。

根据官方的说明

  1. 创建 API Token 。权限选 All Zones,得到 TOKEN
  2. 验证 API 是否 ok
    curl - X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
    -H "Authorization: Bearer TOKEN" \
    -H "Content-Type:application/json"
  3. 获取 :zone_identifier。找那个长得像 ID 的
    curl - X GET "https://api.cloudflare.com/client/v4/zones"\
    -H "Authorization: Bearer TOKEN" \
    -H "Content-Type:application/json"
  4. 因为脑残的 RESTful 风格 API 无法实现 upsert,先创建个DNS。得到 :identifier
    curl -svk -X POST "https://api.cloudflare.com/client/v4/zones/:zone_identifier/dns_records"\
    -H "Authorization: Bearer TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"type":"A","name":"my_ddns.est.im","content":"'$(ifstatus wan | jsonfilter -e '@["ipv4-address"][0].address')'","ttl":600,"proxied":false}'
  5. 日常更新 DNS
    curl -svk -X PUT "https://api.cloudflare.com/client/v4/zones/:zone_identifier/dns_records/:identifier" \
    -H "Authorization: Bearer TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"type":"A","name":"my_ddns.est.im","content":"'$(ifstatus wan | jsonfilter -e '@["ipv4-address"][0].address')'","ttl":600,"proxied":false}'

感觉挺麻烦。而且没法 API 网关自动把调用者的 IP 设为 A 记录。但是聊胜于无吧。高版本的 OpenWRT /usr/lib/ddns/services
直接支持 Cloudflare 了不用这么麻烦了。

Posted

stdout

nsq-py, pynsq, gnsq compared

nsq is a lightweight message queue like Kafka/RabbitMQ/RocketMQ/RedisQueue/Celery.

For python bindings there ware three libraries out there:

  • pynsq: the official build, but requires tornado
  • gnsq: the gevent only consumer/publisher, might have race condition in state machine. Don't use.
  • nsq-py: select() based, but compatible with gevent/tornado/threading+select.

So far nsq-py looks reall promising because it's compatible with more options with concurrency, but when I used it with massive TCP connections to nsqd, gevent threw "ValueError: filedescriptor out of range in select()" error.

Upon further investigation, it was a select(2) limitation. The kernel will limit max tcp fileno 1024.

See CPython source of selectmodule.c:

      if (!_PyIsSelectable_fd(v)) {
          PyErr_SetString(PyExc_ValueError,
                      "filedescriptor out of range in select()");
          goto finally;
      }

And _PyIsSelectable_fd is defined as

  #define _PyIsSelectable_fd(FD) ((unsigned int)(FD) < (unsigned int)FD_SETSIZE)

And the const FD_SETSIZE is described at man page:

The Linux kernel imposes no fixed limit, but the glibc implementation makes fd_set a fixed- size type, with FD_SETSIZE defined as 1024, and the FD_*() macros operating according to that limit. To monitor file descriptors greater than 1023, use poll(2) or epoll(7) instead.

Lessons I learned today: don't use libraries based on select.select() if you have too many connections. Even with Gevent's monkey patch, select() wont block on each eventlet, but still it will not allow fileno greater or equal to 1024. Sad.

Posted

stdout