Mercurial > hg > pnewss

annotate fetchnews.php @ 14:372f4e195986 draft default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added article blacklist function, switch to binary-safe latin1 character set for database
author Ivo Smits <Ivo@UCIS.nl>
date Thu, 17 Jul 2014 23:24:17 +0200
parents cccd73f72bf6
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
1 #!/usr/bin/php
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
2 <?php
8
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
3 /* Copyright 2010 Ivo Smits <Ivo@UCIS.nl>. All rights reserved.
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
4 Redistribution and use in source and binary forms, with or without modification, are
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
5 permitted provided that the following conditions are met:
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
6
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
7 1. Redistributions of source code must retain the above copyright notice, this list of
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
8 conditions and the following disclaimer.
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
9
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
10 2. Redistributions in binary form must reproduce the above copyright notice, this list
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
11 of conditions and the following disclaimer in the documentation and/or other materials
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
12 provided with the distribution.
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
13
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
14 THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
15 WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
16 FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
17 CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
18 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
19 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
20 ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
21 NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
22 ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
23
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
24 The views and conclusions contained in the software and documentation are those of the
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
25 authors and should not be interpreted as representing official policies, either expressed
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
26 or implied, of Ivo Smits.*/
005339a1b2ce Added copyright notice
Ivo Smits <Ivo@UCIS.nl>
parents: 7
diff changeset
27
2
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
28 chdir(__DIR__);
3
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
29 require_once './common.php';
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
30
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
31 foreach ($db->evalAllAssoc('SELECT * FROM `peers`') as $peer) {
12
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
32 if (!$peer['enabled']) continue;
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
33 $socket = stream_socket_client($peer['address']);
11
e0807e0b1a67 Added common.php, updated server synchronization to continue after a connection failure
Ivo Smits <Ivo@UCIS.nl>
parents: 9
diff changeset
34 if ($socket === FALSE) {
e0807e0b1a67 Added common.php, updated server synchronization to continue after a connection failure
Ivo Smits <Ivo@UCIS.nl>
parents: 9
diff changeset
35 print("Could not connect to peer $peer[address]\n");
e0807e0b1a67 Added common.php, updated server synchronization to continue after a connection failure
Ivo Smits <Ivo@UCIS.nl>
parents: 9
diff changeset
36 continue;
e0807e0b1a67 Added common.php, updated server synchronization to continue after a connection failure
Ivo Smits <Ivo@UCIS.nl>
parents: 9
diff changeset
37 }
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
38 $line = nntp_readline($socket);
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
39 $code = strtok($line, " \t");
2
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
40 if ($code == 200) {
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
41 } else if ($code == 201) {
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
42 $peer['post'] = 0;
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
43 } else die("Error code $code from $peer[address]\n");
12
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
44 while ($peer['post'] == 2) {
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
45 if ($peer['lastposted'] === NULL) {
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
46 $articles = $db->evalAllAssoc('SELECT * FROM `messages` ORDER BY `id` ASC LIMIT 10');
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
47 } else {
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
48 $articles = $db->evalAllAssoc('SELECT * FROM `messages` WHERE `id` > ? ORDER BY `id` ASC LIMIT 10', $peer['lastposted']);
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
49 }
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
50 if (!count($articles)) break;
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
51 foreach ($articles as $article) {
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
52 nntp_writeline($socket, 'IHAVE <'.$article['messageid'].'>');
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
53 $line = nntp_readline($socket);
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
54 $code = strtok($line, " \t");
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
55 if ($code == 435) { //Duplicate
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
56 } else if ($code == 335) { //Please send
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
57 foreach (explode("\r\n", $article['header']) as $line) nntp_writeline_data($socket, $line);
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
58 nntp_writeline($socket, '');
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
59 foreach (explode("\r\n", $article['body']) as $line) nntp_writeline_data($socket, $line);
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
60 nntp_writeline($socket, '.');
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
61 $line = nntp_readline($socket);
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
62 $code = strtok($line, " \t");
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
63 if ($code != 240 && $code != 235) print("Article $article[messageid] was not accepted ($code)\n");
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
64 } else {
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
65 print("IHAVE rejected by remote server, falling back to POST\n");
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
66 $peer['post'] = 1;
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
67 break;
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
68 }
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
69 if ($article['id'] > $peer['lastposted']) $peer['lastposted'] = $article['id'];
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
70 }
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
71 $db->update('UPDATE `peers` SET `lastposted` = ? WHERE `id` = ?', array($peer['lastposted'], $peer['id']));
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
72 }
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
73 nntp_writeline($socket, 'MODE READER');
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
74 $line = nntp_readline($socket);
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
75 foreach ($db->evalAllAssoc('SELECT * FROM `peergroups` WHERE `peer` = ?', $peer['id']) as $peergroup) {
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
76 $group = $db->evalRowAssoc('SELECT * FROM `groups` WHERE `id` = ?', $peergroup['group']);
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
77 nntp_writeline($socket, 'GROUP '.$group['name']);
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
78 $line = nntp_readline($socket);
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
79 $code = strtok($line, " \t");
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
80 if ($code != 211) die("Error code $code from $peer[address]\n");
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
81 strtok(" \t"); //number of articles
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
82 $low = strtok(" \t");
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
83 $high = strtok(" \t");
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
84 strtok(" \t"); //group name
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
85 if ($low != $peergroup['low'] || $high != $peergroup['high'] || $peergroup['low'] === NULL || $peergroup['high'] === NULL) {
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
86 for ($i = $low; $i <= $high; $i++) {
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
87 if ($i >= $peergroup['low'] && $i <= $peergroup['high'] && $peergroup['low'] !== NULL && $peergroup['high'] !== NULL) continue;
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
88 nntp_writeline($socket, 'STAT '.$i);
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
89 $line = nntp_readline($socket);
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
90 $code = strtok($line, " \t");
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
91 if ($code == 423) {
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
92 print("Gap in article numbering at $i\n");
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
93 continue;
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
94 }
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
95 if ($code != 223) die("Error code $code from $peer[address]\n");
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
96 strtok(" \t"); //article number
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
97 $messageid = strtok(" \t");
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
98 if ($messageid[0] != '<' || $messageid[strlen($messageid)-1] != '>') die("Malformed message ID $messageid\n");
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
99 $messageid = substr($messageid, 1, -1);
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
100 $message = $db->evalRowAssoc('SELECT * FROM `messages` WHERE `messageid` = ?', $messageid);
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
101 if ($message) {
13
cccd73f72bf6 Added filtering support and several bugfixes
Ivo Smits <Ivo@UCIS.nl>
parents: 12
diff changeset
102 } else if (!nntp_article_wanted($messageid)) {
cccd73f72bf6 Added filtering support and several bugfixes
Ivo Smits <Ivo@UCIS.nl>
parents: 12
diff changeset
103 print("Ignoring filtered article $messageid\n");
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
104 } else {
3
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
105 nntp_writeline($socket, 'ARTICLE '.$i);
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
106 $line = nntp_readline($socket);
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
107 $code = strtok($line, " \t");
3
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
108 if ($code != 220) die("Error code $code from $peer[address]\n");
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
109 strtok(" \t"); //article number
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
110 $lines = nntp_readlines($socket);
3
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
111 try {
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
112 nntp_article_store($lines);
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
113 } catch (Exception $ex) {
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
114 writelog($ex->getMessage());
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
115 }
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
116 }
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
117 }
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
118 $db->update('UPDATE `peergroups` SET `low` = ?, `high` = ? WHERE `peer` = ? AND `group` = ?', array($low, $high, $peergroup['peer'], $peergroup['group']));
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
119 }
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
120 }
12
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
121 while ($peer['post'] == 1) {
2
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
122 if ($peer['lastposted'] === NULL) {
12
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
123 $articles = $db->evalAllAssoc('SELECT * FROM `messages` ORDER BY `id` ASC LIMIT 5');
2
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
124 } else {
12
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
125 $articles = $db->evalAllAssoc('SELECT * FROM `messages` WHERE `id` > ? ORDER BY `id` ASC LIMIT 5', $peer['lastposted']);
2
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
126 }
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
127 if (!count($articles)) break;
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
128 foreach ($articles as $article) {
12
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
129 nntp_writeline($socket, 'STAT <'.$article['messageid'].'>');
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
130 $line = nntp_readline($socket);
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
131 $code = strtok($line, " \t");
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
132 if ($code == 501) { //Argument error
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
133 print("STAT rejected by remote server, skipping message\n");
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
134 } elseif ($code == 223) { //Exists
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
135 } elseif ($code == 430) { //Not found
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
136 nntp_writeline($socket, 'POST');
9
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
137 $line = nntp_readline($socket);
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
138 $code = strtok($line, " \t");
12
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
139 if ($code != 340) die("Error code $code from $peer[address]\n");
9
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
140 foreach (explode("\r\n", $article['header']) as $line) nntp_writeline_data($socket, $line);
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
141 nntp_writeline($socket, '');
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
142 foreach (explode("\r\n", $article['body']) as $line) nntp_writeline_data($socket, $line);
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
143 nntp_writeline($socket, '.');
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
144 $line = nntp_readline($socket);
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
145 $code = strtok($line, " \t");
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
146 if ($code != 240 && $code != 235) print("Article $article[messageid] was not accepted ($code)\n");
12
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
147 } else {
7917bd536187 Added hook for new articles, detect send/write failures, fixed handling of multiline headers, add Date header if it doesn't exist, add option to disable peers, fixes for synchronization with INN, added streaming mode support, small fixes
Ivo Smits <Ivo@UCIS.nl>
parents: 11
diff changeset
148 die("Error code $code from $peer[address]\n");
9
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
149 }
2
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
150 if ($article['id'] > $peer['lastposted']) $peer['lastposted'] = $article['id'];
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
151 }
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
152 $db->update('UPDATE `peers` SET `lastposted` = ? WHERE `id` = ?', array($peer['lastposted'], $peer['id']));
40e545510a57 Added support for POSTing to the server, added readme and todo, added support for upstream synchronization using POST
Ivo Smits <Ivo@UCIS.nl>
parents: 1
diff changeset
153 }
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
154 nntp_writeline($socket, 'QUIT');
9
ae0c67d72670 Bugfix in server POST code, add support for client side IHAVE and fall back to STAT-before-POST
Ivo Smits <Ivo@UCIS.nl>
parents: 8
diff changeset
155 nntp_readline($socket);
0
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
156 fclose($socket);
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
157 }
d7ab68b71c74 Initial commit
Ivo Smits <Ivo@UCIS.nl>
parents:
diff changeset
158
3
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
159 function writelog($line) {
0dcdb73cbcbf Increased article body length to 16MB in database scheme, cleaned up the code, added a script to forcefully reload an individual article
Ivo Smits <Ivo@UCIS.nl>
parents: 2
diff changeset
160 print($line."\n");
1
61fac319ca3e Update Path and Xref header fields while fetching messages, added database schema export
Ivo Smits <Ivo@UCIS.nl>
parents: 0
diff changeset
161 }